pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedPATCH: logical_work_mem and logical streaming of large in-progress transactions
536+ messages / 26 participants
[nested] [flat]
* PATCH: logical_work_mem and logical streaming of large in-progress transactions
@ 2017-12-23 04:57 Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 14:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erikjan Rijkers <er@xs4all.nl>
2017-12-24 04:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Craig Ringer <craig@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 20:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-11 19:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-02 22:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
0 siblings, 6 replies; 536+ messages in thread
From: Tomas Vondra @ 2017-12-23 04:57 UTC (permalink / raw)
To: pgsql-hackers
Hi all,
Attached is a patch series that implements two features to the logical
replication - ability to define a memory limit for the reorderbuffer
(responsible for building the decoded transactions), and ability to
stream large in-progress transactions (exceeding the memory limit).
I'm submitting those two changes together, because one builds on the
other, and it's beneficial to discuss them together.
PART 1: adding logical_work_mem memory limit (0001)
---------------------------------------------------
Currently, limiting the amount of memory consumed by logical decoding is
tricky (or you might say impossible) for several reasons:
* The value is hard-coded, so it's not quite possible to customize it.
* The amount of decoded changes to keep in memory is restricted by
number of changes. It's not very unclear how this relates to memory
consumption, as the change size depends on table structure, etc.
* The number is "per (sub)transaction", so a transaction with many
subtransactions may easily consume significant amount of memory without
actually hitting the limit.
So the patch does two things. Firstly, it introduces logical_work_mem, a
GUC restricting memory consumed by all transactions currently kept in
the reorder buffer.
Secondly, it adds a simple memory accounting by tracking the amount of
memory used in total (for the whole reorder buffer, to compare against
logical_work_mem) and per transaction (so that we can quickly pick
transaction to spill to disk).
The one wrinkle on the patch is that the memory limit can't be enforced
when reading changes spilled to disk - with multiple subtransactions, we
can't easily predict how many changes to pre-read for each of them. At
that point we still use the existing max_changes_in_memory limit.
Luckily, changes introduced in the other parts of the patch should allow
addressing this deficiency.
PART 2: streaming of large in-progress transactions (0002-0006)
---------------------------------------------------------------
Note: This part is split into multiple smaller chunks, addressing
different parts of the logical decoding infrastructure. That's mostly to
allow easier reviews, though. Ultimately, it's just one patch.
Processing large transactions often results in significant apply lag,
for a couple of reasons. One reason is network bandwidth - while we do
decode the changes incrementally (as we read the WAL), we keep them
locally, either in memory, or spilled to files. Then at commit time, all
the changes get sent to the downstream (and applied) at the same time.
For large transactions the time to do the network transfer may be
significant, causing apply lag.
This patch extends the logical replication infrastructure (output plugin
API, reorder buffer, pgoutput, replication protocol etc.) so allow
streaming of in-progress transactions instead of spilling them to local
files.
The extensions to the API are pretty straightforward. Aside from adding
methods to stream changes/messages and commit a streamed transaction,
the API needs a function to abort a streamed (sub)transaction, and
functions to demarcate a block of streamed changes.
To decode a transaction, we need to know all it's subtransactions, and
invalidations. Currently, those are only known at commit time (although
some assignments may be known earlier), but invalidations are only ever
written in the commit record.
So far that was fine, because we only decode/replay transactions at
commit time, when all of this is known (because it's either in commit
record, or written before it).
But for in-progress transactions (i.e. the subject of interest here),
that is not the case. So the patch modifies WAL-logging to ensure those
two bits of information are written immediately (for wal_level=logical).
For assignments that was fairly simple, thanks to existing caching. For
invalidations, it requires a new WAL record type and a couple of changes
in inval.c.
On the apply side, we simply receive the streamed changes, write them
into a file (one file for toplevel transaction, which is possible thanks
to the assignments being known immediately). And then at commit time the
changes are replayed locally, without having to copy a large chunk of
data over network.
WAL overhead
------------
Of course, these changes to WAL logging are not for free - logging
assignments individually (instead of multiple subtransactions at once)
means higher xlog record overhead. Similarly, (sub)transactions doing a
lot of DDL may result in a lot of invalidations written to WAL (again,
with full xlog record overhead per invalidation).
I've done a number of tests to measure the impact, and for extreme
corner cases the additional amount of WAL is about 40% in both cases.
By an "extreme corner case" I mean a workloads intentionally triggering
many assignments/invalidations, without doing a lot of meaningful work.
For assignments, imagine a single-row table (no indexes), and a
transaction like this one:
BEGIN;
UPDATE t SET v = v + 1;
SAVEPOINT s1;
UPDATE t SET v = v + 1;
SAVEPOINT s2;
UPDATE t SET v = v + 1;
SAVEPOINT s3;
...
UPDATE t SET v = v + 1;
SAVEPOINT s10;
UPDATE t SET v = v + 1;
COMMIT;
For invalidations, add a CREATE TEMPORARY TABLE to each subtransaction.
For more realistic workloads (large table with indexes, runs long enough
to generate FPIs, etc.) the overhead drops below 5%. Which is much more
acceptable, of course, although not perfect.
In both cases, there was pretty much no measurable impact on performance
(as measured by tps).
I do not think there's a way around this requirement (having assignments
and invalidations), if we want to decode in-progress transactions. But
perhaps it would be possible to do some sort of caching (say, at command
level), to reduce the xlog record overhead? Not sure.
All ideas are welcome, of course. In the worst case, I think we can add
a GUC enabling this additional logging - when disabled, streaming of
in-progress transactions would not be possible.
Simplifying ReorderBuffer
-------------------------
One interesting consequence of having assignments is that we could get
rid of the ReorderBuffer iterator, used to merge changes from subxacts.
The assignments allow us to keep changes for each toplevel transaction
in a single list, in LSN order, and just walk it. Abort can be performed
by remembering position of the first change in each subxact, and just
discarding the tail.
This is what the apply worker does with the streamed changes and aborts.
It would also allow us to enforce the memory limit while restoring
transactions spilled to disk, because we would not have the problem with
restoring changes for many subtransactions.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-me.patch.gz (application/gzip, 6.8K) download
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical.patch.gz (application/gzip, 2.7K) download
0003-Issue-individual-invalidations-with-wal_level-logica.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer.patch.gz (application/gzip, 10.6K) download
0006-Add-support-for-streaming-to-built-in-replication.patch.gz (application/gzip, 13.3K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2017-12-23 14:03 ` Erikjan Rijkers <er@xs4all.nl>
2017-12-23 20:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
5 siblings, 1 reply; 536+ messages in thread
From: Erikjan Rijkers @ 2017-12-23 14:03 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
On 2017-12-23 05:57, Tomas Vondra wrote:
> Hi all,
>
> Attached is a patch series that implements two features to the logical
> replication - ability to define a memory limit for the reorderbuffer
> (responsible for building the decoded transactions), and ability to
> stream large in-progress transactions (exceeding the memory limit).
>
logical replication of 2 instances is OK but 3 and up fail with:
TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
"reorderbuffer.c", Line: 1773)
I can cobble up a script but I hope you have enough from the assertion
to see what's going wrong...
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 14:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erikjan Rijkers <er@xs4all.nl>
@ 2017-12-23 20:06 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 22:23 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2017-12-23 20:06 UTC (permalink / raw)
To: Erikjan Rijkers <er@xs4all.nl>; +Cc: pgsql-hackers
On 12/23/2017 03:03 PM, Erikjan Rijkers wrote:
> On 2017-12-23 05:57, Tomas Vondra wrote:
>> Hi all,
>>
>> Attached is a patch series that implements two features to the logical
>> replication - ability to define a memory limit for the reorderbuffer
>> (responsible for building the decoded transactions), and ability to
>> stream large in-progress transactions (exceeding the memory limit).
>>
>
> logical replication of 2 instances is OK but 3 and up fail with:
>
> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
> "reorderbuffer.c", Line: 1773)
>
> I can cobble up a script but I hope you have enough from the assertion
> to see what's going wrong...
The assertion says that the iterator produces changes in order that does
not correlate with LSN. But I have a hard time understanding how that
could happen, particularly because according to the line number this
happens in ReorderBufferCommit(), i.e. the current (non-streaming) case.
So instructions to reproduce the issue would be very helpful.
Attached is v2 of the patch series, fixing two bugs I discovered today.
I don't think any of these is related to your issue, though.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch.gz (application/gzip, 6.8K) download
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch.gz (application/gzip, 2.7K) download
0003-Issue-individual-invalidations-with-wal_level-log-v2.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch.gz (application/gzip, 10.6K) download
0006-Add-support-for-streaming-to-built-in-replication-v2.patch.gz (application/gzip, 13.9K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 14:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erikjan Rijkers <er@xs4all.nl>
2017-12-23 20:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2017-12-23 22:23 ` Erik Rijkers <er@xs4all.nl>
2017-12-24 00:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2017-12-23 22:23 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
On 2017-12-23 21:06, Tomas Vondra wrote:
> On 12/23/2017 03:03 PM, Erikjan Rijkers wrote:
>> On 2017-12-23 05:57, Tomas Vondra wrote:
>>> Hi all,
>>>
>>> Attached is a patch series that implements two features to the
>>> logical
>>> replication - ability to define a memory limit for the reorderbuffer
>>> (responsible for building the decoded transactions), and ability to
>>> stream large in-progress transactions (exceeding the memory limit).
>>>
>>
>> logical replication of 2 instances is OK but 3 and up fail with:
>>
>> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
>> "reorderbuffer.c", Line: 1773)
>>
>> I can cobble up a script but I hope you have enough from the assertion
>> to see what's going wrong...
>
> The assertion says that the iterator produces changes in order that
> does
> not correlate with LSN. But I have a hard time understanding how that
> could happen, particularly because according to the line number this
> happens in ReorderBufferCommit(), i.e. the current (non-streaming)
> case.
>
> So instructions to reproduce the issue would be very helpful.
Using:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch
0003-Issue-individual-invalidations-with-wal_level-log-v2.patch
0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch
0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch
0006-Add-support-for-streaming-to-built-in-replication-v2.patch
As you expected the problem is the same with these new patches.
I have now tested more, and seen that it not always fails. I guess that
it here fails 3 times out of 4. But the laptop I'm using at the moment
is old and slow -- it may well be a factor as we've seen before [1].
Attached is the bash that I put together. I tested with
NUM_INSTANCES=2, which yields success, and NUM_INSTANCES=3, which fails
often. This same program run with HEAD never seems to fail (I tried a
few dozen times).
thanks,
Erik Rijkers
[1]
https://www.postgresql.org/message-id/3897361c7010c4ac03f358173adbcd60%40xs4all.nl
Attachments:
test.sh (text/x-shellscript, 7.3K) download
#!/bin/bash
unset PGSERVICE PGSERVICEFILE PGDATA PGPORT PGDATABASE
# PGPASSFILE must be set and have the appropriate entries
env | grep ^PG
PROJECT=large_logical
# PROJECT=HEAD
BIN_DIR=$HOME/pg_stuff/pg_installations/pgsql.$PROJECT/bin
POSTGRES=$BIN_DIR/postgres
INITDB=$BIN_DIR/initdb
TMP_DIR=$HOME'/tmp/'$PROJECT
devel_file=${TMP_DIR}'/.devel'
NUM_INSTANCES=3
BASE_PORT=6015 # ports 6015, 6016, 6017
port1=$(( $BASE_PORT + 0 ))
port2=$(( $port1 + 1 ))
port3=$(( $port1 + 2 ))
scale=1 dbname=postgres pubname=pub1 subname=sub1
if [[ ! -d $TMP_DIR ]]; then mkdir $TMP_DIR; fi
echo 's3kr1t' > $devel_file
max_wal_senders=10 # publication side
max_replication_slots=10 # publication side and subscription side
max_worker_processes=12 # subscription side
max_logical_replication_workers=10 # subscription side
max_sync_workers_per_subscription=4 # subscription side
for n in `seq 1 $NUM_INSTANCES`; do
port=$(( $BASE_PORT + $n -1 ))
data_dir=$TMP_DIR/pgsql.instance${n}/data
server_dir=$TMP_DIR/pgsql.instance${n}
$INITDB --pgdata=$data_dir --encoding=UTF8 --auth=scram-sha-256 --pwfile=$devel_file # --waldir=$xlog_dir
( $POSTGRES -D $data_dir -p $port \
--wal_level=logical \
--max_replication_slots=$max_replication_slots \
--max_worker_processes=$max_worker_processes \
--max_logical_replication_workers=$max_logical_replication_workers \
--max_wal_senders=$max_wal_senders \
--max_sync_workers_per_subscription=$max_sync_workers_per_subscription \
--logging_collector=on \
--log_directory=${server_dir} \
--log_filename=logfile.${port} \
--log_replication_commands=on \
--autovacuum=off & )
# --logical_work_mem=128MB & )
# pg_isready -d $dbname --timeout=60 -p $port
done
#sleep $NUM_INSTANCES
#pg_isready -d $dbname -qp 6015 --timeout=60
#pg_isready -d $dbname -qp 6016 --timeout=60
num_loop=$(( $NUM_INSTANCES - 1 ))
$BIN_DIR/pgbench --port=$BASE_PORT --quiet --initialize --scale=$scale $dbname
echo "alter table pgbench_history add column hid serial primary key" | $BIN_DIR/psql -d $dbname -p $BASE_PORT -X
#pg_isready -d $dbname -qp 6015 --timeout=60
#pg_isready -d $dbname -qp 6016 --timeout=60
for n in `seq 1 $num_loop`; do
target_port=$(( $BASE_PORT + $n ))
pg_dump -Fc -p $BASE_PORT \
--exclude-table-data=pgbench_history --exclude-table-data=pgbench_accounts \
--exclude-table-data=pgbench_branches --exclude-table-data=pgbench_tellers \
-tpgbench_history -tpgbench_accounts \
-tpgbench_branches -tpgbench_tellers \
$dbname | pg_restore -1 -p $target_port -d $dbname
done
#echo "sleep 2 (after dump/restore)"; sleep 2
for n in `seq 1 $num_loop`; do
pubport=$(( $BASE_PORT + $n - 1 ))
subport=$(( $BASE_PORT + $n ))
appname='casc:'${subport}'<'${pubport}
echo "create publication $pubname for all tables" | psql -d $dbname -p $pubport -X
echo "create subscription $subname
connection 'port=${pubport} dbname=${dbname} application_name=${appname}'
publication $pubname with (enabled=false, slot_name=${subname}_${subport});" | psql -d $dbname -p $subport -X
echo "alter subscription $subname enable; " | psql -d $dbname -p $subport -X
done
c_a1=$( echo "select count(*) from pgbench_accounts"|psql -d$dbname -qtAX -p$port1)
c_b1=$( echo "select count(*) from pgbench_branches"|psql -d$dbname -qtAX -p$port1)
c_t1=$( echo "select count(*) from pgbench_tellers "|psql -d$dbname -qtAX -p$port1)
c_h1=$( echo "select count(*) from pgbench_history "|psql -d$dbname -qtAX -p$port1)
mda1=$( echo "select aid,bid,abalance,filler from pgbench_accounts order by aid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
mdb1=$( echo "select bid,bbalance,filler from pgbench_branches order by bid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
mdt1=$( echo "select tid,bid,tbalance,filler from pgbench_tellers order by tid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
mdh1=$( echo "select hid,bid,aid,delta,mtime,filler,hid from pgbench_history order by hid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
md5_1=$(echo "$mda1 $mdb1 $mdt1 $mdh1" | md5sum | cut -b 1-7 )
printf "a,b,t,h $port1 %6d %6d %6d %6d $mda1 $mdb1 $mdt1 $mdh1 $md5_1\n" $c_a1 $c_b1 $c_t1 $c_h1
ver1=$( echo "select substring(version(),1,70)" | psql -d $dbname -qtAXp $port1 )
ver2=$( echo "select substring(version(),1,70)" | psql -d $dbname -qtAXp $port2 )
if [[ $NUM_INSTANCES -gt 2 ]]; then
ver3=$( echo "select substring(version(),1,70)" | psql -d $dbname -qtAXp $port3 )
fi
echo
rc=0
while [[ $rc -eq 0 ]]
do
mda2=$(echo "select aid,bid,abalance,filler from pgbench_accounts order by aid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
mdb2=$(echo "select bid,bbalance,filler from pgbench_branches order by bid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
mdt2=$(echo "select tid,bid,tbalance,filler from pgbench_tellers order by tid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
mdh2=$(echo "select hid,bid,aid,delta,mtime,filler,hid from pgbench_history order by hid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
c_a2=$(echo "select count(*) from pgbench_accounts"|psql -d$dbname -qtAX -p$port2)
c_b2=$(echo "select count(*) from pgbench_branches"|psql -d$dbname -qtAX -p$port2)
c_t2=$(echo "select count(*) from pgbench_tellers "|psql -d$dbname -qtAX -p$port2)
c_h2=$(echo "select count(*) from pgbench_history "|psql -d$dbname -qtAX -p$port2)
md5_2=$(echo "$mda2 $mdb2 $mdt2 $mdh2" | md5sum | cut -b 1-7 )
if [[ $NUM_INSTANCES -gt 2 ]]; then
mda3=$(echo "select aid,bid,abalance,filler from pgbench_accounts order by aid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
mdb3=$(echo "select bid,bbalance,filler from pgbench_branches order by bid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
mdt3=$(echo "select tid,bid,tbalance,filler from pgbench_tellers order by tid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
mdh3=$(echo "select hid,bid,aid,delta,mtime,filler,hid from pgbench_history order by hid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
c_a3=$(echo "select count(*) from pgbench_accounts"|psql -d$dbname -qtAX -p$port3)
c_b3=$(echo "select count(*) from pgbench_branches"|psql -d$dbname -qtAX -p$port3)
c_t3=$(echo "select count(*) from pgbench_tellers "|psql -d$dbname -qtAX -p$port3)
c_h3=$(echo "select count(*) from pgbench_history "|psql -d$dbname -qtAX -p$port3)
md5_3=$(echo "$mda3 $mdb3 $mdt3 $mdh3" | md5sum | cut -b 1-7 )
fi
echo "-- $POSTGRES"
printf "a,b,t,h $port1 %7d %6d %6d %6d $mda1 $mdb1 $mdt1 $mdh1 $md5_1 " $c_a1 $c_b1 $c_t1 $c_h1; echo "$ver1"
printf "a,b,t,h $port2 %7d %6d %6d %6d $mda2 $mdb2 $mdt2 $mdh2 $md5_2 " $c_a2 $c_b2 $c_t2 $c_h2; echo "$ver2"
if [[ $NUM_INSTANCES -gt 2 ]]; then
printf "a,b,t,h $port3 %7d %6d %6d %6d $mda3 $mdb3 $mdt3 $mdh3 $md5_3 " $c_a3 $c_b3 $c_t3 $c_h3; echo "$ver3"
fi
if [[ $NUM_INSTANCES -eq 2 ]]; then
if [[ "$md5_1" == "$md5_2" ]] ; then echo "OK - done."; break; fi
elif [[ $NUM_INSTANCES -eq 3 ]]; then
if [[ "$md5_1" == "$md5_3" ]] ; then echo "OK - done."; break; fi
fi
sleep 1
rc=$?
done
for n in `seq 1 $NUM_INSTANCES`; do
port=$(( $BASE_PORT + $n -1 ))
data_dir=$TMP_DIR/pgsql.instance${n}/data
$BIN_DIR/pg_ctl stop -w -D $data_dir
done
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 14:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erikjan Rijkers <er@xs4all.nl>
2017-12-23 20:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 22:23 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2017-12-24 00:42 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-24 09:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2017-12-24 00:42 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: pgsql-hackers
On 12/23/2017 11:23 PM, Erik Rijkers wrote:
> On 2017-12-23 21:06, Tomas Vondra wrote:
>> On 12/23/2017 03:03 PM, Erikjan Rijkers wrote:
>>> On 2017-12-23 05:57, Tomas Vondra wrote:
>>>> Hi all,
>>>>
>>>> Attached is a patch series that implements two features to the logical
>>>> replication - ability to define a memory limit for the reorderbuffer
>>>> (responsible for building the decoded transactions), and ability to
>>>> stream large in-progress transactions (exceeding the memory limit).
>>>>
>>>
>>> logical replication of 2 instances is OK but 3 and up fail with:
>>>
>>> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
>>> "reorderbuffer.c", Line: 1773)
>>>
>>> I can cobble up a script but I hope you have enough from the assertion
>>> to see what's going wrong...
>>
>> The assertion says that the iterator produces changes in order that does
>> not correlate with LSN. But I have a hard time understanding how that
>> could happen, particularly because according to the line number this
>> happens in ReorderBufferCommit(), i.e. the current (non-streaming) case.
>>
>> So instructions to reproduce the issue would be very helpful.
>
> Using:
>
> 0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch
> 0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch
> 0003-Issue-individual-invalidations-with-wal_level-log-v2.patch
> 0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch
> 0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch
> 0006-Add-support-for-streaming-to-built-in-replication-v2.patch
>
> As you expected the problem is the same with these new patches.
>
> I have now tested more, and seen that it not always fails. I guess that
> it here fails 3 times out of 4. But the laptop I'm using at the moment
> is old and slow -- it may well be a factor as we've seen before [1].
>
> Attached is the bash that I put together. I tested with
> NUM_INSTANCES=2, which yields success, and NUM_INSTANCES=3, which fails
> often. This same program run with HEAD never seems to fail (I tried a
> few dozen times).
>
Thanks. Unfortunately I still can't reproduce the issue. I even tried
running it in valgrind, to see if there are some memory access issues
(which should also slow it down significantly).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 14:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erikjan Rijkers <er@xs4all.nl>
2017-12-23 20:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 22:23 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2017-12-24 00:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2017-12-24 09:00 ` Erik Rijkers <er@xs4all.nl>
2017-12-25 17:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2017-12-24 09:00 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
>>>>
>>>> logical replication of 2 instances is OK but 3 and up fail with:
>>>>
>>>> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
>>>> "reorderbuffer.c", Line: 1773)
>>>>
>>>> I can cobble up a script but I hope you have enough from the
>>>> assertion
>>>> to see what's going wrong...
>>>
>>> The assertion says that the iterator produces changes in order that
>>> does
>>> not correlate with LSN. But I have a hard time understanding how that
>>> could happen, particularly because according to the line number this
>>> happens in ReorderBufferCommit(), i.e. the current (non-streaming)
>>> case.
>>>
>>> So instructions to reproduce the issue would be very helpful.
>>
>> Using:
>>
>> 0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch
>> 0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch
>> 0003-Issue-individual-invalidations-with-wal_level-log-v2.patch
>> 0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch
>> 0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch
>> 0006-Add-support-for-streaming-to-built-in-replication-v2.patch
>>
>> As you expected the problem is the same with these new patches.
>>
>> I have now tested more, and seen that it not always fails. I guess
>> that
>> it here fails 3 times out of 4. But the laptop I'm using at the
>> moment
>> is old and slow -- it may well be a factor as we've seen before [1].
>>
>> Attached is the bash that I put together. I tested with
>> NUM_INSTANCES=2, which yields success, and NUM_INSTANCES=3, which
>> fails
>> often. This same program run with HEAD never seems to fail (I tried a
>> few dozen times).
>>
>
> Thanks. Unfortunately I still can't reproduce the issue. I even tried
> running it in valgrind, to see if there are some memory access issues
> (which should also slow it down significantly).
One wonders again if 2ndquadrant shouldn't invest in some old hardware
;)
Another Good Thing would be if there was a provision in the buildfarm to
test patches like these.
But I'm probably not to first one to suggest that; no doubt it'll be
possible someday. In the meantime I'll try to repeat this crash on
other machines (but that will be after the holidays).
Erik Rijkers
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 14:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erikjan Rijkers <er@xs4all.nl>
2017-12-23 20:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 22:23 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2017-12-24 00:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-24 09:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2017-12-25 17:40 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-25 22:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2017-12-25 17:40 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: pgsql-hackers
On 12/24/2017 10:00 AM, Erik Rijkers wrote:
>>>>>
>>>>> logical replication of 2 instances is OK but 3 and up fail with:
>>>>>
>>>>> TRAP: FailedAssertion("!(last_lsn < change->lsn)", File:
>>>>> "reorderbuffer.c", Line: 1773)
>>>>>
>>>>> I can cobble up a script but I hope you have enough from the assertion
>>>>> to see what's going wrong...
>>>>
>>>> The assertion says that the iterator produces changes in order that
>>>> does
>>>> not correlate with LSN. But I have a hard time understanding how that
>>>> could happen, particularly because according to the line number this
>>>> happens in ReorderBufferCommit(), i.e. the current (non-streaming)
>>>> case.
>>>>
>>>> So instructions to reproduce the issue would be very helpful.
>>>
>>> Using:
>>>
>>> 0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v2.patch
>>> 0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v2.patch
>>> 0003-Issue-individual-invalidations-with-wal_level-log-v2.patch
>>> 0004-Extend-the-output-plugin-API-with-stream-methods-v2.patch
>>> 0005-Implement-streaming-mode-in-ReorderBuffer-v2.patch
>>> 0006-Add-support-for-streaming-to-built-in-replication-v2.patch
>>>
>>> As you expected the problem is the same with these new patches.
>>>
>>> I have now tested more, and seen that it not always fails. I guess that
>>> it here fails 3 times out of 4. But the laptop I'm using at the moment
>>> is old and slow -- it may well be a factor as we've seen before [1].
>>>
>>> Attached is the bash that I put together. I tested with
>>> NUM_INSTANCES=2, which yields success, and NUM_INSTANCES=3, which fails
>>> often. This same program run with HEAD never seems to fail (I tried a
>>> few dozen times).
>>>
>>
>> Thanks. Unfortunately I still can't reproduce the issue. I even tried
>> running it in valgrind, to see if there are some memory access issues
>> (which should also slow it down significantly).
>
> One wonders again if 2ndquadrant shouldn't invest in some old hardware ;)
>
Well, I've done tests on various machines, including some really slow
ones, and I still haven't managed to reproduce the failures using your
script. So I don't think that would really help. But I have reproduced
it by using a custom stress test script.
Turns out the asserts are overly strict - instead of
Assert(prev_lsn < current_lsn);
it should have been
Assert(prev_lsn <= current_lsn);
because some XLOG records may contain multiple rows (e.g. MULTI_INSERT).
The attached v3 fixes this issue, and also a couple of other thinkos:
1) The AssertChangeLsnOrder assert check was somewhat broken.
2) We've been sending aborts for all subtransactions, even those not yet
streamed. So downstream got confused and fell over because of an assert.
3) The streamed transactions were written to /tmp, using filenames using
subscription OID and XID of the toplevel transaction. That's fine, as
long as there's just a single replica running - if there are more, the
filenames will clash, causing really strange failures. So move the files
to base/pgsql_tmp where regular temporary files are written. I'm not
claiming this is perfect, perhaps we need to invent another location.
FWIW I believe the relation sync cache is somewhat broken by the
streaming. I thought resetting it would be good enough, but it's more
complicated (and trickier) than that. I'm aware of it, and I'll look
into that next - but probably not before 2018.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Introduce-logical_work_mem-to-limit-ReorderBuffer-v3.patch.gz (application/gzip, 6.8K) download
0002-Issue-XLOG_XACT_ASSIGNMENT-with-wal_level-logical-v3.patch.gz (application/gzip, 2.7K) download
0003-Issue-individual-invalidations-with-wal_level-log-v3.patch.gz (application/gzip, 4.8K) download
0004-Extend-the-output-plugin-API-with-stream-methods-v3.patch.gz (application/gzip, 5.3K) download
0005-Implement-streaming-mode-in-ReorderBuffer-v3.patch.gz (application/gzip, 10.7K) download
0006-Add-support-for-streaming-to-built-in-replication-v3.patch.gz (application/gzip, 14.0K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 14:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erikjan Rijkers <er@xs4all.nl>
2017-12-23 20:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 22:23 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2017-12-24 00:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-24 09:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2017-12-25 17:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2017-12-25 22:08 ` Erik Rijkers <er@xs4all.nl>
2017-12-26 14:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dmitry Dolgov <9erthalion6@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2017-12-25 22:08 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
That indeed fixed the problem: running that same pgbench test, I see no
crashes anymore (on any of 3 different machines, and with several
pgbench parameters).
Thank you,
Erik Rijkers
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 14:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erikjan Rijkers <er@xs4all.nl>
2017-12-23 20:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 22:23 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2017-12-24 00:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-24 09:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2017-12-25 17:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-25 22:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2017-12-26 14:50 ` Dmitry Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Dmitry Dolgov @ 2017-12-26 14:50 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
> On 25 December 2017 at 18:40, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:
> The attached v3 fixes this issue, and also a couple of other thinkos
Thank you for the patch, it looks quite interesting. After a quick look at
it
(mostly the first one so far, but I'm going to continue) I have a few
questions:
> + * XXX With many subtransactions this might be quite slow, because we'll
have
> + * to walk through all of them. There are some options how we could
improve
> + * that: (a) maintain some secondary structure with transactions sorted
by
> + * amount of changes, (b) not looking for the entirely largest
transaction,
> + * but e.g. for transaction using at least some fraction of the memory
limit,
> + * and (c) evicting multiple transactions at once, e.g. to free a given
portion
> + * of the memory limit (e.g. 50%).
Do you want to address these possible alternatives somehow in this patch or
you
want to left it outside? Maybe it makes sense to apply some combination of
them, e.g. maintain a secondary structure with relatively large
transactions,
and then start evicting them. If it's somehow not enough, then start to
evict
multiple transactions at once (option "c").
> + /*
> + * We clamp manually-set values to at least 64kB. The
maintenance_work_mem
> + * uses a higher minimum value (1MB), so this is OK.
> + */
> + if (*newval < 64)
> + *newval = 64;
> +
I'm not sure what's recommended practice here, but maybe it makes sense to
have a warning here about changing this value to 64kB? Otherwise it can be
unexpected.
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2017-12-24 04:51 ` Craig Ringer <craig@2ndquadrant.com>
2017-12-24 13:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
5 siblings, 1 reply; 536+ messages in thread
From: Craig Ringer @ 2017-12-24 04:51 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: pgsql-hackers
On 23 December 2017 at 12:57, Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:
> Hi all,
>
> Attached is a patch series that implements two features to the logical
> replication - ability to define a memory limit for the reorderbuffer
> (responsible for building the decoded transactions), and ability to
> stream large in-progress transactions (exceeding the memory limit).
>
> I'm submitting those two changes together, because one builds on the
> other, and it's beneficial to discuss them together.
>
>
> PART 1: adding logical_work_mem memory limit (0001)
> ---------------------------------------------------
>
> Currently, limiting the amount of memory consumed by logical decoding is
> tricky (or you might say impossible) for several reasons:
>
> * The value is hard-coded, so it's not quite possible to customize it.
>
> * The amount of decoded changes to keep in memory is restricted by
> number of changes. It's not very unclear how this relates to memory
> consumption, as the change size depends on table structure, etc.
>
> * The number is "per (sub)transaction", so a transaction with many
> subtransactions may easily consume significant amount of memory without
> actually hitting the limit.
>
Also, even without subtransactions, we assemble a ReorderBufferTXN per
transaction. Since transactions usually occur concurrently, systems with
many concurrent txns can face lots of memory use.
We can't exclude tables that won't actually be replicated at the reorder
buffering phase either. So txns use memory whether or not they do anything
interesting as far as a given logical decoding session is concerned. Even
if we'll throw all the data away we must buffer and assemble it first so we
can make that decision.
Because logical decoding considers snapshots and cid increments even from
other DBs (at least when the txn makes catalog changes) the memory use can
get BIG too. I was recently working with a system that had accumulated 2GB
of snapshots ... on each slot. With 7 slots, one for each DB.
So there's lots of room for difficulty with unpredictable memory use.
So the patch does two things. Firstly, it introduces logical_work_mem, a
> GUC restricting memory consumed by all transactions currently kept in
> the reorder buffer
>
Does this consider the (currently high, IIRC) overhead of tracking
serialized changes?
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-24 04:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Craig Ringer <craig@2ndquadrant.com>
@ 2017-12-24 13:43 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2017-12-24 13:43 UTC (permalink / raw)
To: Craig Ringer <craig@2ndquadrant.com>; +Cc: pgsql-hackers
On 12/24/2017 05:51 AM, Craig Ringer wrote:
> On 23 December 2017 at 12:57, Tomas Vondra <tomas.vondra@2ndquadrant.com
> <mailto:tomas.vondra@2ndquadrant.com>> wrote:
>
> Hi all,
>
> Attached is a patch series that implements two features to the logical
> replication - ability to define a memory limit for the reorderbuffer
> (responsible for building the decoded transactions), and ability to
> stream large in-progress transactions (exceeding the memory limit).
>
> I'm submitting those two changes together, because one builds on the
> other, and it's beneficial to discuss them together.
>
>
> PART 1: adding logical_work_mem memory limit (0001)
> ---------------------------------------------------
>
> Currently, limiting the amount of memory consumed by logical decoding is
> tricky (or you might say impossible) for several reasons:
>
> * The value is hard-coded, so it's not quite possible to customize it.
>
> * The amount of decoded changes to keep in memory is restricted by
> number of changes. It's not very unclear how this relates to memory
> consumption, as the change size depends on table structure, etc.
>
> * The number is "per (sub)transaction", so a transaction with many
> subtransactions may easily consume significant amount of memory without
> actually hitting the limit.
>
>
> Also, even without subtransactions, we assemble a ReorderBufferTXN
> per transaction. Since transactions usually occur concurrently,
> systems with many concurrent txns can face lots of memory use.
>
I don't see how that could be a problem, considering the number of
toplevel transactions is rather limited (to max_connections or so).
> We can't exclude tables that won't actually be replicated at the reorder
> buffering phase either. So txns use memory whether or not they do
> anything interesting as far as a given logical decoding session is
> concerned. Even if we'll throw all the data away we must buffer and
> assemble it first so we can make that decision.
Yep.
> Because logical decoding considers snapshots and cid increments even
> from other DBs (at least when the txn makes catalog changes) the memory
> use can get BIG too. I was recently working with a system that had
> accumulated 2GB of snapshots ... on each slot. With 7 slots, one for
> each DB.
>
> So there's lots of room for difficulty with unpredictable memory use.
>
Yep.
> So the patch does two things. Firstly, it introduces logical_work_mem, a
> GUC restricting memory consumed by all transactions currently kept in
> the reorder buffer
>
>
> Does this consider the (currently high, IIRC) overhead of tracking
> serialized changes?
>
Consider in what sense?
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2018-01-02 15:07 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
5 siblings, 1 reply; 536+ messages in thread
From: Peter Eisentraut @ 2018-01-02 15:07 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On 12/22/17 23:57, Tomas Vondra wrote:
> PART 1: adding logical_work_mem memory limit (0001)
> ---------------------------------------------------
The documentation in this patch contains some references to later
features (streaming). Perhaps that could be separated so that the
patches can be applied independently.
I don't see the need to tie this setting to maintenance_work_mem.
maintenance_work_mem is often set to very large values, which could then
have undesirable side effects on this use.
Moreover, the name logical_work_mem makes it sound like it's a logical
version of work_mem. Maybe we could think of another name.
I think we need a way to report on how much memory is actually used, so
the setting can be tuned. Something analogous to log_temp_files perhaps.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2018-01-03 19:53 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2018-01-03 19:53 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 01/02/2018 04:07 PM, Peter Eisentraut wrote:
> On 12/22/17 23:57, Tomas Vondra wrote:
>> PART 1: adding logical_work_mem memory limit (0001)
>> ---------------------------------------------------
>
> The documentation in this patch contains some references to later
> features (streaming). Perhaps that could be separated so that the
> patches can be applied independently.
>
Yeah, that's probably a good idea. But now that you mention it, I wonder
if "streaming" is really a good term. We already use it for "streaming
replication" and it may be quite confusing to use it for another feature
(particularly when it's streaming within logical streaming replication).
But I can't really think of a better name ...
> I don't see the need to tie this setting to maintenance_work_mem.
> maintenance_work_mem is often set to very large values, which could
> then have undesirable side effects on this use.
>
Well, we need to pick some default value, and we can either use a fixed
value (not sure what would be a good default) or tie it to an existing
GUC. We only really have work_mem and maintenance_work_mem, and the
walsender process will never use more than one such buffer. Which seems
to be closer to maintenance_work_mem.
Pretty much any default value can have undesirable side effects.
> Moreover, the name logical_work_mem makes it sound like it's a logical
> version of work_mem. Maybe we could think of another name.
>
I won't object to a better name, of course. Any proposals?
> I think we need a way to report on how much memory is actually used,
> so the setting can be tuned. Something analogous to log_temp_files
> perhaps.
>
Yes, I agree. I'm just about to submit an updated version of the patch
series, that also introduces a few columns pg_stat_replication, tracking
this type of stats (amount of data spilled to disk or streamed, etc.).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2018-01-09 02:24 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Peter Eisentraut @ 2018-01-09 02:24 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On 1/3/18 14:53, Tomas Vondra wrote:
>> I don't see the need to tie this setting to maintenance_work_mem.
>> maintenance_work_mem is often set to very large values, which could
>> then have undesirable side effects on this use.
>
> Well, we need to pick some default value, and we can either use a fixed
> value (not sure what would be a good default) or tie it to an existing
> GUC. We only really have work_mem and maintenance_work_mem, and the
> walsender process will never use more than one such buffer. Which seems
> to be closer to maintenance_work_mem.
>
> Pretty much any default value can have undesirable side effects.
Let's just make it an independent setting unless we know any better. We
don't have a lot of settings that depend on other settings, and the ones
we do have a very specific relationship.
>> Moreover, the name logical_work_mem makes it sound like it's a logical
>> version of work_mem. Maybe we could think of another name.
>
> I won't object to a better name, of course. Any proposals?
logical_decoding_[work_]mem?
>> I think we need a way to report on how much memory is actually used,
>> so the setting can be tuned. Something analogous to log_temp_files
>> perhaps.
>
> Yes, I agree. I'm just about to submit an updated version of the patch
> series, that also introduces a few columns pg_stat_replication, tracking
> this type of stats (amount of data spilled to disk or streamed, etc.).
That seems OK. Perhaps we could bring forward the part of that patch
that applies to this feature.
That would also help testing *this* feature and determine what
appropriate settings are.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2019-09-27 09:03 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-09-27 09:03 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 9, 2018 at 7:55 AM Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
>
> On 1/3/18 14:53, Tomas Vondra wrote:
> >> I don't see the need to tie this setting to maintenance_work_mem.
> >> maintenance_work_mem is often set to very large values, which could
> >> then have undesirable side effects on this use.
> >
> > Well, we need to pick some default value, and we can either use a fixed
> > value (not sure what would be a good default) or tie it to an existing
> > GUC. We only really have work_mem and maintenance_work_mem, and the
> > walsender process will never use more than one such buffer. Which seems
> > to be closer to maintenance_work_mem.
> >
> > Pretty much any default value can have undesirable side effects.
>
> Let's just make it an independent setting unless we know any better. We
> don't have a lot of settings that depend on other settings, and the ones
> we do have a very specific relationship.
>
> >> Moreover, the name logical_work_mem makes it sound like it's a logical
> >> version of work_mem. Maybe we could think of another name.
> >
> > I won't object to a better name, of course. Any proposals?
>
> logical_decoding_[work_]mem?
>
Having a separate variable for this can give more flexibility, but
OTOH it will add one more knob which user might not have a good idea
to set. What are the problems we see if directly use work_mem for
this case?
If we can't use work_mem, then I think the name proposed by you
(logical_decoding_work_mem) sounds good to me.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-09-27 11:25 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-09-27 11:25 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Sep 27, 2019 at 02:33:32PM +0530, Amit Kapila wrote:
>On Tue, Jan 9, 2018 at 7:55 AM Peter Eisentraut
><peter.eisentraut@2ndquadrant.com> wrote:
>>
>> On 1/3/18 14:53, Tomas Vondra wrote:
>> >> I don't see the need to tie this setting to maintenance_work_mem.
>> >> maintenance_work_mem is often set to very large values, which could
>> >> then have undesirable side effects on this use.
>> >
>> > Well, we need to pick some default value, and we can either use a fixed
>> > value (not sure what would be a good default) or tie it to an existing
>> > GUC. We only really have work_mem and maintenance_work_mem, and the
>> > walsender process will never use more than one such buffer. Which seems
>> > to be closer to maintenance_work_mem.
>> >
>> > Pretty much any default value can have undesirable side effects.
>>
>> Let's just make it an independent setting unless we know any better. We
>> don't have a lot of settings that depend on other settings, and the ones
>> we do have a very specific relationship.
>>
>> >> Moreover, the name logical_work_mem makes it sound like it's a logical
>> >> version of work_mem. Maybe we could think of another name.
>> >
>> > I won't object to a better name, of course. Any proposals?
>>
>> logical_decoding_[work_]mem?
>>
>
>Having a separate variable for this can give more flexibility, but
>OTOH it will add one more knob which user might not have a good idea
>to set. What are the problems we see if directly use work_mem for
>this case?
>
IMHO it's similar to autovacuum_work_mem - we have an independent
setting, but most people use it as -1 so we use maintenance_work_mem as
a default value. I think it makes sense to do the same thing here.
It does ad an extra knob anyway (I don't think we should just use
maintenance_work_mem directly, the user should have an option to
override it when needed). But most users will not notice.
FWIW I don't think we should use work_mem, maintenace_work_mem seems
somewhat more appropriate here (not related to queries, etc.).
>If we can't use work_mem, then I think the name proposed by you
>(logical_decoding_work_mem) sounds good to me.
>
Yes, that name seems better.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2019-09-28 08:06 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-09-28 08:06 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Sep 27, 2019 at 4:55 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Fri, Sep 27, 2019 at 02:33:32PM +0530, Amit Kapila wrote:
> >On Tue, Jan 9, 2018 at 7:55 AM Peter Eisentraut
> ><peter.eisentraut@2ndquadrant.com> wrote:
> >>
> >> On 1/3/18 14:53, Tomas Vondra wrote:
> >> >> I don't see the need to tie this setting to maintenance_work_mem.
> >> >> maintenance_work_mem is often set to very large values, which could
> >> >> then have undesirable side effects on this use.
> >> >
> >> > Well, we need to pick some default value, and we can either use a fixed
> >> > value (not sure what would be a good default) or tie it to an existing
> >> > GUC. We only really have work_mem and maintenance_work_mem, and the
> >> > walsender process will never use more than one such buffer. Which seems
> >> > to be closer to maintenance_work_mem.
> >> >
> >> > Pretty much any default value can have undesirable side effects.
> >>
> >> Let's just make it an independent setting unless we know any better. We
> >> don't have a lot of settings that depend on other settings, and the ones
> >> we do have a very specific relationship.
> >>
> >> >> Moreover, the name logical_work_mem makes it sound like it's a logical
> >> >> version of work_mem. Maybe we could think of another name.
> >> >
> >> > I won't object to a better name, of course. Any proposals?
> >>
> >> logical_decoding_[work_]mem?
> >>
> >
> >Having a separate variable for this can give more flexibility, but
> >OTOH it will add one more knob which user might not have a good idea
> >to set. What are the problems we see if directly use work_mem for
> >this case?
> >
>
> IMHO it's similar to autovacuum_work_mem - we have an independent
> setting, but most people use it as -1 so we use maintenance_work_mem as
> a default value. I think it makes sense to do the same thing here.
>
> It does ad an extra knob anyway (I don't think we should just use
> maintenance_work_mem directly, the user should have an option to
> override it when needed). But most users will not notice.
>
> FWIW I don't think we should use work_mem, maintenace_work_mem seems
> somewhat more appropriate here (not related to queries, etc.).
>
I have the same concern for using maintenace_work_mem as Peter E.
which is that the value of maintenace_work_mem will generally be
higher which is suitable for its current purpose, but not for the
purpose this patch is using. AFAIU, at this stage we want a better
memory accounting system for logical decoding and we are not sure what
is a good value for this variable. So, I think using work_mem or
maintenace_work_mem should serve the purpose. Later, if we have
requirements from people to have better control over the memory
required for this purpose then we can introduce a new variable.
I understand that currently work_mem is primarily tied with memory
used for query workspaces, but it might be okay to extend it for this
purpose. Another point is that the default for that sound to be more
appealing for this case. I can see the argument against it which is
having a separate variable will make the things look clean and give
better control. So, if we can't convince ourselves for using
work_mem, we can introduce a new guc variable and keep the default as
4MB or work_mem.
I feel it is always tempting to introduce a new guc for the different
tasks unless there is an exact match, but OTOH, having lesser guc's
has its own advantage which is that people don't have to bother about
a new setting which they need to tune and especially for which they
can't decide with ease. I am not telling that we should not introduce
new guc when it is required, but just to give more thought before
doing so.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-09-28 19:09 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-09-28 19:09 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Sep 28, 2019 at 01:36:46PM +0530, Amit Kapila wrote:
>On Fri, Sep 27, 2019 at 4:55 PM Tomas Vondra
><tomas.vondra@2ndquadrant.com> wrote:
>>
>> On Fri, Sep 27, 2019 at 02:33:32PM +0530, Amit Kapila wrote:
>> >On Tue, Jan 9, 2018 at 7:55 AM Peter Eisentraut
>> ><peter.eisentraut@2ndquadrant.com> wrote:
>> >>
>> >> On 1/3/18 14:53, Tomas Vondra wrote:
>> >> >> I don't see the need to tie this setting to maintenance_work_mem.
>> >> >> maintenance_work_mem is often set to very large values, which could
>> >> >> then have undesirable side effects on this use.
>> >> >
>> >> > Well, we need to pick some default value, and we can either use a fixed
>> >> > value (not sure what would be a good default) or tie it to an existing
>> >> > GUC. We only really have work_mem and maintenance_work_mem, and the
>> >> > walsender process will never use more than one such buffer. Which seems
>> >> > to be closer to maintenance_work_mem.
>> >> >
>> >> > Pretty much any default value can have undesirable side effects.
>> >>
>> >> Let's just make it an independent setting unless we know any better. We
>> >> don't have a lot of settings that depend on other settings, and the ones
>> >> we do have a very specific relationship.
>> >>
>> >> >> Moreover, the name logical_work_mem makes it sound like it's a logical
>> >> >> version of work_mem. Maybe we could think of another name.
>> >> >
>> >> > I won't object to a better name, of course. Any proposals?
>> >>
>> >> logical_decoding_[work_]mem?
>> >>
>> >
>> >Having a separate variable for this can give more flexibility, but
>> >OTOH it will add one more knob which user might not have a good idea
>> >to set. What are the problems we see if directly use work_mem for
>> >this case?
>> >
>>
>> IMHO it's similar to autovacuum_work_mem - we have an independent
>> setting, but most people use it as -1 so we use maintenance_work_mem as
>> a default value. I think it makes sense to do the same thing here.
>>
>> It does ad an extra knob anyway (I don't think we should just use
>> maintenance_work_mem directly, the user should have an option to
>> override it when needed). But most users will not notice.
>>
>> FWIW I don't think we should use work_mem, maintenace_work_mem seems
>> somewhat more appropriate here (not related to queries, etc.).
>>
>
>I have the same concern for using maintenace_work_mem as Peter E.
>which is that the value of maintenace_work_mem will generally be
>higher which is suitable for its current purpose, but not for the
>purpose this patch is using. AFAIU, at this stage we want a better
>memory accounting system for logical decoding and we are not sure what
>is a good value for this variable. So, I think using work_mem or
>maintenace_work_mem should serve the purpose. Later, if we have
>requirements from people to have better control over the memory
>required for this purpose then we can introduce a new variable.
>
>I understand that currently work_mem is primarily tied with memory
>used for query workspaces, but it might be okay to extend it for this
>purpose. Another point is that the default for that sound to be more
>appealing for this case. I can see the argument against it which is
>having a separate variable will make the things look clean and give
>better control. So, if we can't convince ourselves for using
>work_mem, we can introduce a new guc variable and keep the default as
>4MB or work_mem.
>
>I feel it is always tempting to introduce a new guc for the different
>tasks unless there is an exact match, but OTOH, having lesser guc's
>has its own advantage which is that people don't have to bother about
>a new setting which they need to tune and especially for which they
>can't decide with ease. I am not telling that we should not introduce
>new guc when it is required, but just to give more thought before
>doing so.
>
I do think having a separate GUC is a must, irrespectedly of what other
GUC (if any) is used as a default. You're right the maintenance_work_mem
value might be too high (e.g. in cases with many subscriptions), but the
same issue applies to work_mem - there's no guarantee work_mem is lower
than maintenance_work_mem, and in analytics databases it may be set very
high. So work_mem does not really solve the issue
IMHO we can't really do without a new GUC. It's not difficult to create
examples that would benefit from small/large memory limit, depending on
the number of subscriptions etc.
I do however agree the GUC does not have to be tied to any existing one,
it was just an attempt to use a more sensible default value. I do think
m_w_m would be fine, but I can live with using an explicit value.
So that's what I did in the attached patch - I've renamed the GUC to
logical_decoding_work_mem, detached it from m_w_m and set the default to
64MB (i.e. the same default as m_w_m). It should also fix all the issues
from the recent reviews (at least I believe so).
I've realized that one of the subsequent patches allows overriding the
limit for individual subscriptions (in the CREATE SUBSCRIPTION command).
I think it'd be good to move this bit forward, but I think it can be
done in a separate patch.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.patch.gz (application/gzip, 10.2K) download
0002-Immediately-WAL-log-assignments.patch.gz (application/gzip, 6.6K) download
0003-Issue-individual-invalidations-with-wal_level-logica.patch.gz (application/gzip, 4.9K) download
0004-Extend-the-output-plugin-API-with-stream-methods.patch.gz (application/gzip, 6.1K) download
0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structure.patch.gz (application/gzip, 2.5K) download
0006-Support-decoding-of-two-phase-transactions-at-PREPAR.patch.gz (application/gzip, 9.0K) download
0007-Gracefully-handle-concurrent-aborts-of-uncommitted.patch.gz (application/gzip, 4.4K) download
0008-Teach-test_decoding-plugin-to-work-with-2PC.patch.gz (application/gzip, 5.8K) download
0009-Implement-streaming-mode-in-ReorderBuffer.patch.gz (application/gzip, 12.6K) download
0010-Add-support-for-streaming-to-built-in-replication.patch.gz (application/gzip, 19.8K) download
0011-Track-statistics-for-streaming-spilling.patch.gz (application/gzip, 4.2K) download
0012-Enable-streaming-for-all-subscription-TAP-tests.patch.gz (application/gzip, 2.3K) download
0013-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch.gz (application/gzip, 628B) download
0014-Add-TAP-test-for-streaming-vs.-DDL.patch.gz (application/gzip, 1.5K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2019-09-29 05:54 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-29 17:30 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2019-09-29 05:54 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Sat, Sep 28, 2019 at 01:36:46PM +0530, Amit Kapila wrote:
> >On Fri, Sep 27, 2019 at 4:55 PM Tomas Vondra
> ><tomas.vondra@2ndquadrant.com> wrote:
>
> I do think having a separate GUC is a must, irrespectedly of what other
> GUC (if any) is used as a default. You're right the maintenance_work_mem
> value might be too high (e.g. in cases with many subscriptions), but the
> same issue applies to work_mem - there's no guarantee work_mem is lower
> than maintenance_work_mem, and in analytics databases it may be set very
> high. So work_mem does not really solve the issue
>
> IMHO we can't really do without a new GUC. It's not difficult to create
> examples that would benefit from small/large memory limit, depending on
> the number of subscriptions etc.
>
> I do however agree the GUC does not have to be tied to any existing one,
> it was just an attempt to use a more sensible default value. I do think
> m_w_m would be fine, but I can live with using an explicit value.
>
> So that's what I did in the attached patch - I've renamed the GUC to
> logical_decoding_work_mem, detached it from m_w_m and set the default to
> 64MB (i.e. the same default as m_w_m).
Fair enough, let's not argue more on this unless someone else wants to
share his opinion.
> It should also fix all the issues
> from the recent reviews (at least I believe so).
>
Have you given any thought on creating a test case for this patch? I
think you also told that you will test some worst-case scenarios and
report the numbers so that we are convinced that the current eviction
algorithm is good.
> I've realized that one of the subsequent patches allows overriding the
> limit for individual subscriptions (in the CREATE SUBSCRIPTION command).
> I think it'd be good to move this bit forward, but I think it can be
> done in a separate patch.
>
Yeah, it is better to deal it separately as I am also not entirely
convinced at this stage about this parameter. I have mentioned the
same in the previous email as well.
While glancing through the changes, I noticed a small thing:
+#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use maintenance_work_mem
I guess this need to be updated.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-09-29 17:30 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-29 17:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Alvaro Herrera @ 2019-09-29 17:30 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2019-Sep-29, Amit Kapila wrote:
> On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
> > So that's what I did in the attached patch - I've renamed the GUC to
> > logical_decoding_work_mem, detached it from m_w_m and set the default to
> > 64MB (i.e. the same default as m_w_m).
>
> Fair enough, let's not argue more on this unless someone else wants to
> share his opinion.
I just read this part of the conversation and I agree that having a
separate GUC with its own value independent from other GUCs is a good
solution. Tying it to m_w_m seemed reasonable, but it's true that
people frequently set m_w_m very high, and it would be undesirable to
propagate that value to logical decoding memory usage.
I wonder what would constitute good advice on how to set this value, I
mean what is the metric that the user needs to be thinking about. Is
it the total of memory required to keep all concurrent write transactions
in memory? (Quick example: if you do 2048 wTPS and each transaction
lasts 1s, and each transaction does 1kB of logically-decoded changes,
then ~2MB are sufficient for the average case. Is that correct? I
*think* that full-page images do not count, correct? With these things
in mind users could go through pg_waldump output and figure out what to
set the value to.)
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-29 17:30 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2019-09-29 17:56 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-09-29 17:56 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Sep 29, 2019 at 02:30:44PM -0300, Alvaro Herrera wrote:
>On 2019-Sep-29, Amit Kapila wrote:
>
>> On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
>
>> > So that's what I did in the attached patch - I've renamed the GUC to
>> > logical_decoding_work_mem, detached it from m_w_m and set the default to
>> > 64MB (i.e. the same default as m_w_m).
>>
>> Fair enough, let's not argue more on this unless someone else wants to
>> share his opinion.
>
>I just read this part of the conversation and I agree that having a
>separate GUC with its own value independent from other GUCs is a good
>solution. Tying it to m_w_m seemed reasonable, but it's true that
>people frequently set m_w_m very high, and it would be undesirable to
>propagate that value to logical decoding memory usage.
>
>
>I wonder what would constitute good advice on how to set this value, I
>mean what is the metric that the user needs to be thinking about. Is
>it the total of memory required to keep all concurrent write transactions
>in memory? (Quick example: if you do 2048 wTPS and each transaction
>lasts 1s, and each transaction does 1kB of logically-decoded changes,
>then ~2MB are sufficient for the average case. Is that correct?
Yes, something like that. Essentially we'd like to keep all concurrent
transactions decoded in memory, to eliminate the need to spill to disk.
One of the subsequent patches adds some subscription-level stats, so
maybe we don't need to worry about this too much - the stats seem like a
better source of information for tuning.
>I *think* that full-page images do not count, correct? With these
>things in mind users could go through pg_waldump output and figure out
>what to set the value to.)
>
Right, FPW do not matter here.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-10-01 13:25 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-10-01 13:25 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Sep 29, 2019 at 11:24 AM Amit Kapila <amit.kapila16@gmail.com>
wrote:
> On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> >
>
> Yeah, it is better to deal it separately as I am also not entirely
> convinced at this stage about this parameter. I have mentioned the
> same in the previous email as well.
>
> While glancing through the changes, I noticed a small thing:
> +#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use
maintenance_work_mem
>
> I guess this need to be updated.
>
On further testing, I found that the patch seems to have problems with
toast. Consider below scenario:
Session-1
Create table large_text(t1 text);
INSERT INTO large_text
SELECT (SELECT string_agg('x', ',')
FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
Session-2
SELECT * FROM pg_create_logical_replication_slot('regression_slot',
'test_decoding');
SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
*--kaboom*
The second statement in Session-2 leads to a crash.
Other than that, I am not sure if the changes related to spill to disk
after logical_decoding_work_mem works for toast table as I couldn't hit
that code for toast table case, but I might be missing something. As
mentioned previously, I feel there should be some way to test whether this
patch works for the cases it claims to work. As of now, I have to check
via debugging. Let me know if there is any way, I can test this.
I am reluctant to say, but I think this patch still needs some more work
(review, test, rework) before we can commit it.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-10-01 13:51 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2019-10-01 13:51 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
>On Sun, Sep 29, 2019 at 11:24 AM Amit Kapila <amit.kapila16@gmail.com>
>wrote:
>> On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra
>> <tomas.vondra@2ndquadrant.com> wrote:
>> >
>>
>> Yeah, it is better to deal it separately as I am also not entirely
>> convinced at this stage about this parameter. I have mentioned the
>> same in the previous email as well.
>>
>> While glancing through the changes, I noticed a small thing:
>> +#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use
>maintenance_work_mem
>>
>> I guess this need to be updated.
>>
>
>On further testing, I found that the patch seems to have problems with
>toast. Consider below scenario:
>Session-1
>Create table large_text(t1 text);
>INSERT INTO large_text
>SELECT (SELECT string_agg('x', ',')
>FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
>
>Session-2
>SELECT * FROM pg_create_logical_replication_slot('regression_slot',
>'test_decoding');
>SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
>*--kaboom*
>
>The second statement in Session-2 leads to a crash.
>
OK, thanks for the report - will investigate.
>Other than that, I am not sure if the changes related to spill to disk
>after logical_decoding_work_mem works for toast table as I couldn't hit
>that code for toast table case, but I might be missing something. As
>mentioned previously, I feel there should be some way to test whether this
>patch works for the cases it claims to work. As of now, I have to check
>via debugging. Let me know if there is any way, I can test this.
>
That's one of the reasons why I proposed to move the statistics (which
say how many transactions / bytes were spilled to disk) from a later
patch in the series. I don't think there's a better way.
>I am reluctant to say, but I think this patch still needs some more work
>(review, test, rework) before we can commit it.
>
I agreee.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2019-10-01 22:57 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-10-01 22:57 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra <tomas.vondra@2ndquadrant.com>
wrote:
> On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
> >
> >On further testing, I found that the patch seems to have problems with
> >toast. Consider below scenario:
> >Session-1
> >Create table large_text(t1 text);
> >INSERT INTO large_text
> >SELECT (SELECT string_agg('x', ',')
> >FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
> >
> >Session-2
> >SELECT * FROM pg_create_logical_replication_slot('regression_slot',
> >'test_decoding');
> >SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
> >*--kaboom*
> >
> >The second statement in Session-2 leads to a crash.
> >
>
> OK, thanks for the report - will investigate.
>
It was an assertion failure in ReorderBufferCleanupTXN at below line:
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
> >Other than that, I am not sure if the changes related to spill to disk
> >after logical_decoding_work_mem works for toast table as I couldn't hit
> >that code for toast table case, but I might be missing something. As
> >mentioned previously, I feel there should be some way to test whether this
> >patch works for the cases it claims to work. As of now, I have to check
> >via debugging. Let me know if there is any way, I can test this.
> >
>
> That's one of the reasons why I proposed to move the statistics (which
> say how many transactions / bytes were spilled to disk) from a later
> patch in the series. I don't think there's a better way.
>
>
I like that idea, but I think you need to split that patch to only get the
stats related to the spill. It would be easier to review if you can
prepare that atop of
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-10-02 22:32 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-03 09:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-14 09:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 3 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-10-02 22:32 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Oct 02, 2019 at 04:27:30AM +0530, Amit Kapila wrote:
>On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra <tomas.vondra@2ndquadrant.com>
>wrote:
>
>> On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
>> >
>> >On further testing, I found that the patch seems to have problems with
>> >toast. Consider below scenario:
>> >Session-1
>> >Create table large_text(t1 text);
>> >INSERT INTO large_text
>> >SELECT (SELECT string_agg('x', ',')
>> >FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
>> >
>> >Session-2
>> >SELECT * FROM pg_create_logical_replication_slot('regression_slot',
>> >'test_decoding');
>> >SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
>> >*--kaboom*
>> >
>> >The second statement in Session-2 leads to a crash.
>> >
>>
>> OK, thanks for the report - will investigate.
>>
>
>It was an assertion failure in ReorderBufferCleanupTXN at below line:
>+ /* Check we're not mixing changes from different transactions. */
>+ Assert(change->txn == txn);
>
Can you still reproduce this issue with the patch I sent on 28/9? I have
been unable to trigger the failure, and it seems pretty similar to the
failure you reported (and I fixed) on 28/9.
>> >Other than that, I am not sure if the changes related to spill to disk
>> >after logical_decoding_work_mem works for toast table as I couldn't hit
>> >that code for toast table case, but I might be missing something. As
>> >mentioned previously, I feel there should be some way to test whether this
>> >patch works for the cases it claims to work. As of now, I have to check
>> >via debugging. Let me know if there is any way, I can test this.
>> >
>>
>> That's one of the reasons why I proposed to move the statistics (which
>> say how many transactions / bytes were spilled to disk) from a later
>> patch in the series. I don't think there's a better way.
>>
>>
>I like that idea, but I think you need to split that patch to only get the
>stats related to the spill. It would be easier to review if you can
>prepare that atop of
>0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.
>
Sure, I wasn't really proposing to adding all stats from that patch,
including those related to streaming. We need to extract just those
related to spilling. And yes, it needs to be moved right after 0001.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2019-10-03 07:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-10 10:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2 siblings, 3 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-10-03 07:48 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
I have attempted to test the performance of (Stream + Spill) vs
(Stream + BGW pool) and I can see the similar gain what Alexey had
shown[1].
In addition to this, I have rebased the latest patchset [2] without
the two-phase logical decoding patch set.
Test results:
I have repeated the same test as Alexy[1] for 1kk and 1kk data and
here is my result
Stream + Spill
N time on master(sec) Total xact time (sec)
1kk 6 21
3kk 18 55
Stream + BGW pool
N time on master(sec) Total xact time (sec)
1kk 6 13
3kk 19 35
Patch details:
All the patches are the same as posted on [2] except
1. 0006-Gracefully-handle-concurrent-aborts-of-uncommitted -> I have
removed the handling of error which is specific for 2PC
2. 0007-Implement-streaming-mode-in-ReorderBuffer -> Rebased without 2PC
3. 0009-Extend-the-concurrent-abort-handling-for-in-progress -> New
patch to handle concurrent abort error for the in-progress transaction
and also add handling for the sub transaction's abort.
4. v3-0014-BGWorkers-pool-for-streamed-transactions-apply -> Rebased
Alexey's patch
[1] https://www.postgresql.org/message-id/8eda5118-2dd0-79a1-4fe9-eec7e334de17%40postgrespro.ru
[2] https://www.postgresql.org/message-id/20190928190917.hrpknmq76v3ts3lj%40development
On Thu, Oct 3, 2019 at 4:03 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Wed, Oct 02, 2019 at 04:27:30AM +0530, Amit Kapila wrote:
> >On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra <tomas.vondra@2ndquadrant.com>
> >wrote:
> >
> >> On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote:
> >> >
> >> >On further testing, I found that the patch seems to have problems with
> >> >toast. Consider below scenario:
> >> >Session-1
> >> >Create table large_text(t1 text);
> >> >INSERT INTO large_text
> >> >SELECT (SELECT string_agg('x', ',')
> >> >FROM generate_series(1, 1000000)) FROM generate_series(1, 1000);
> >> >
> >> >Session-2
> >> >SELECT * FROM pg_create_logical_replication_slot('regression_slot',
> >> >'test_decoding');
> >> >SELECT * FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL);
> >> >*--kaboom*
> >> >
> >> >The second statement in Session-2 leads to a crash.
> >> >
> >>
> >> OK, thanks for the report - will investigate.
> >>
> >
> >It was an assertion failure in ReorderBufferCleanupTXN at below line:
> >+ /* Check we're not mixing changes from different transactions. */
> >+ Assert(change->txn == txn);
> >
>
> Can you still reproduce this issue with the patch I sent on 28/9? I have
> been unable to trigger the failure, and it seems pretty similar to the
> failure you reported (and I fixed) on 28/9.
>
> >> >Other than that, I am not sure if the changes related to spill to disk
> >> >after logical_decoding_work_mem works for toast table as I couldn't hit
> >> >that code for toast table case, but I might be missing something. As
> >> >mentioned previously, I feel there should be some way to test whether this
> >> >patch works for the cases it claims to work. As of now, I have to check
> >> >via debugging. Let me know if there is any way, I can test this.
> >> >
> >>
> >> That's one of the reasons why I proposed to move the statistics (which
> >> say how many transactions / bytes were spilled to disk) from a later
> >> patch in the series. I don't think there's a better way.
> >>
> >>
> >I like that idea, but I think you need to split that patch to only get the
> >stats related to the spill. It would be easier to review if you can
> >prepare that atop of
> >0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.
> >
>
> Sure, I wasn't really proposing to adding all stats from that patch,
> including those related to streaming. We need to extract just those
> related to spilling. And yes, it needs to be moved right after 0001.
>
> regards
>
> --
> Tomas Vondra http://www.2ndQuadrant.com
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>
>
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer.patch (application/octet-stream, 33.6K) download
From d2830f660ca86dd84b5c377748d133cfe57463ae Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:04:54 +0200
Subject: [PATCH 01/13] Add logical_decoding_work_mem to limit ReorderBuffer
memory usage
Instead of deciding to serialize a transaction merely based on the
number of changes in that xact (toplevel or subxact), this makes
the decisions based on amount of memory consumed by the changes.
The memory limit is defined by a new logical_decoding_work_mem GUC,
so for example we can do this
SET logical_decoding_work_mem = '128kB'
to trigger very aggressive streaming. The minimum value is 64kB.
When adding a change to a transaction, we account for the size in
two places. Firstly, in the ReorderBuffer, which is then used to
decide if we reached the total memory limit. And secondly in the
transaction the change belongs to, so that we can pick the largest
transaction to evict (and serialize to disk).
We still use max_changes_in_memory when loading changes serialized
to disk. The trouble is we can't use the memory limit directly as
there might be multiple subxact serialized, we need to read all of
them but we don't know how many are there (and which subxact to
read first).
We do not serialize the ReorderBufferTXN entries, so if there is a
transaction with many subxacts, most memory may be in this type of
objects. Those records are not included in the memory accounting.
We also do not account for INTERNAL_TUPLECID changes, which are
kept in a separate list and not evicted from memory. Transactions
with many CTID changes may consume significant amounts of memory,
but we can't really do much about that.
The current eviction algorithm is very simple - the transaction is
picked merely by size, while it might be useful to also consider age
(LSN) of the changes for example. With the new Generational memory
allocator, evicting the oldest changes would make it more likely
the memory gets actually pfreed.
The logical_decoding_work_mem may be set either in postgresql.conf,
in which case it serves as the default for all publishers on that
instance, or when creating the subscription, using a work_mem
paramemter in the WITH clause (specifies number of kilobytes).
---
doc/src/sgml/config.sgml | 21 ++
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++-
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/reorderbuffer.c | 292 ++++++++++++++++++++-
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++-
src/backend/utils/misc/guc.c | 36 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/catalog/pg_subscription.h | 3 +
src/include/replication/reorderbuffer.h | 16 ++
src/include/replication/walreceiver.h | 1 +
13 files changed, 441 insertions(+), 20 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 619ac8c..a207ac2 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1715,6 +1715,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index afee283..7e3ba8e 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 2e67a58..d85e831 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -66,7 +66,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -97,6 +98,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -182,6 +185,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -325,6 +338,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -341,7 +356,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -419,6 +434,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -682,10 +703,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -710,6 +734,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -721,7 +752,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -759,7 +791,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -796,7 +828,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a..65b3266 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 8ce28ad..6228140 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -49,6 +49,34 @@
* GenerationContext for the variable-length transaction data (allocated
* and freed in groups with similar lifespan).
*
+ * To limit the amount of memory used by decoded changes, we track memory
+ * used at the reorder buffer level (i.e. total amount of memory), and for
+ * each toplevel transaction. When the total amount of used memory exceeds
+ * the limit, the toplevel transaction consuming the most memory is then
+ * serialized to disk.
+ *
+ * Only decoded changes are evicted from memory (spilled to disk), not the
+ * transaction records. The number of toplevel transactions is limited,
+ * but a transaction with many subtransactions may still consume significant
+ * amounts of memory. The transaction records are fairly small, though, and
+ * are not included in the memory limit.
+ *
+ * The current eviction algorithm is very simple - the transaction is
+ * picked merely by size, while it might be useful to also consider age
+ * (LSN) of the changes for example. With the new Generational memory
+ * allocator, evicting the oldest changes would make it more likely the
+ * memory gets actually freed.
+ *
+ * We still rely on max_changes_in_memory when loading serialized changes
+ * back into memory. At that point we can't use the memory limit directly
+ * as we load the subxacts independently. One option do deal with this
+ * would be to count the subxacts, and allow each to allocate 1/N of the
+ * memory limit. That however does not seem very appealing, because with
+ * many subtransactions it may easily cause trashing (short cycles of
+ * deserializing and applying very few changes). We probably should give
+ * a bit more memory to the oldest subtransactions, because it's likely
+ * the source for the next sequence of changes.
+ *
* -------------------------------------------------------------------------
*/
#include "postgres.h"
@@ -154,7 +182,8 @@ typedef struct ReorderBufferDiskChange
* resource management here, but it's not entirely clear what that would look
* like.
*/
-static const Size max_changes_in_memory = 4096;
+int logical_decoding_work_mem;
+static const Size max_changes_in_memory = 4096; /* XXX for restore only */
/* ---------------------------------------
* primary reorderbuffer support routines
@@ -189,7 +218,7 @@ static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTX
* Disk serialization support functions
* ---------------------------------------
*/
-static void ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferCheckMemoryLimit(ReorderBuffer *rb);
static void ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
int fd, ReorderBufferChange *change);
@@ -217,6 +246,14 @@ static void ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
static void ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
Relation relation, ReorderBufferChange *change);
+/*
+ * ---------------------------------------
+ * memory accounting
+ * ---------------------------------------
+ */
+static Size ReorderBufferChangeSize(ReorderBufferChange *change);
+static void ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change, bool addition);
/*
* Allocate a new ReorderBuffer and clean out any old serialized state from
@@ -269,6 +306,7 @@ ReorderBufferAllocate(void)
buffer->outbuf = NULL;
buffer->outbufsize = 0;
+ buffer->size = 0;
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
@@ -374,6 +412,9 @@ ReorderBufferGetChange(ReorderBuffer *rb)
void
ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
{
+ /* update memory accounting info */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
/* free contained data */
switch (change->action)
{
@@ -585,12 +626,18 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
change->lsn = lsn;
+ change->txn = txn;
+
Assert(InvalidXLogRecPtr != lsn);
dlist_push_tail(&txn->changes, &change->node);
txn->nentries++;
txn->nentries_mem++;
- ReorderBufferCheckSerializeTXN(rb, txn);
+ /* update memory accounting information */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
+
+ /* check the memory limits and evict something if needed */
+ ReorderBufferCheckMemoryLimit(rb);
}
/*
@@ -1217,6 +1264,9 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
change = dlist_container(ReorderBufferChange, node, iter.cur);
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
+
ReorderBufferReturnChange(rb, change);
}
@@ -1229,7 +1279,11 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferChange *change;
change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* Check we're not mixing changes from different transactions. */
+ Assert(change->txn == txn);
Assert(change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID);
+
ReorderBufferReturnChange(rb, change);
}
@@ -2082,9 +2136,48 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
ReorderBufferQueueChange(rb, xid, lsn, change);
}
+/*
+ * Update the memory accounting info. We track memory used by the whole
+ * reorder buffer and the transaction containing the change.
+ */
+static void
+ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
+ ReorderBufferChange *change,
+ bool addition)
+{
+ Size sz;
+
+ Assert(change->txn);
+
+ /*
+ * Ignore tuple CID changes, because those are not evicted when
+ * reaching memory limit. So we just don't count them, because it
+ * might easily trigger a pointless attempt to spill/stream.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
+ return;
+
+ sz = ReorderBufferChangeSize(change);
+
+ if (addition)
+ {
+ change->txn->size += sz;
+ rb->size += sz;
+ }
+ else
+ {
+ Assert((rb->size >= sz) && (change->txn->size >= sz));
+ change->txn->size -= sz;
+ rb->size -= sz;
+ }
+}
/*
* Add new (relfilenode, tid) -> (cmin, cmax) mappings.
+ *
+ * We do not include this change type in memory accounting, because we
+ * keep CIDs in a separate list and do not evict them when reaching
+ * the memory limit.
*/
void
ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
@@ -2230,20 +2323,84 @@ ReorderBufferSerializeReserve(ReorderBuffer *rb, Size sz)
}
/*
- * Check whether the transaction tx should spill its data to disk.
+ * Find the largest transaction (toplevel or subxact) to evict (spill to disk).
+ *
+ * XXX With many subtransactions this might be quite slow, because we'll have
+ * to walk through all of them. There are some options how we could improve
+ * that: (a) maintain some secondary structure with transactions sorted by
+ * amount of changes, (b) not looking for the entirely largest transaction,
+ * but e.g. for transaction using at least some fraction of the memory limit,
+ * and (c) evicting multiple transactions at once, e.g. to free a given portion
+ * of the memory limit (e.g. 50%).
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTXN(ReorderBuffer *rb)
+{
+ HASH_SEQ_STATUS hash_seq;
+ ReorderBufferTXNByIdEnt *ent;
+ ReorderBufferTXN *largest = NULL;
+
+ hash_seq_init(&hash_seq, rb->by_txn);
+ while ((ent = hash_seq_search(&hash_seq)) != NULL)
+ {
+ ReorderBufferTXN *txn = ent->txn;
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
+ * Check whether the logical_decoding_work_mem limit was reached, and if yes
+ * pick the transaction to evict and spill the changes to disk.
+ *
+ * XXX At this point we select just a single (largest) transaction, but
+ * we might also adapt a more elaborate eviction strategy - for example
+ * evicting enough transactions to free certain fraction (e.g. 50%) of
+ * the memory limit.
*/
static void
-ReorderBufferCheckSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
+ ReorderBufferTXN *txn;
+
+ /* bail out if we haven't exceeded the memory limit */
+ if (rb->size < logical_decoding_work_mem * 1024L)
+ return;
+
/*
- * TODO: improve accounting so we cheaply can take subtransactions into
- * account here.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
*/
- if (txn->nentries_mem >= max_changes_in_memory)
- {
- ReorderBufferSerializeTXN(rb, txn);
- Assert(txn->nentries_mem == 0);
- }
+ txn = ReorderBufferLargestTXN(rb);
+
+ ReorderBufferSerializeTXN(rb, txn);
+
+ /*
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ */
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
+
+ /*
+ * And furthermore, evicting the transaction should get us below the
+ * memory limit again - it is not possible that we're still exceeding the
+ * memory limit after evicting the transaction.
+ *
+ * This follows from the simple fact that the selected transaction is at
+ * least as large as the most recent change (which caused us to go over
+ * the memory limit). So by evicting it we're definitely back below the
+ * memory limit.
+ */
+ Assert(rb->size < logical_decoding_work_mem * 1024L);
}
/*
@@ -2513,6 +2670,84 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
+ * Size of a change in memory.
+ */
+static Size
+ReorderBufferChangeSize(ReorderBufferChange *change)
+{
+ Size sz = sizeof(ReorderBufferChange);
+
+ switch (change->action)
+ {
+ /* fall through these, they're all similar enough */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ {
+ ReorderBufferTupleBuf *oldtup,
+ *newtup;
+ Size oldlen = 0;
+ Size newlen = 0;
+
+ oldtup = change->data.tp.oldtuple;
+ newtup = change->data.tp.newtuple;
+
+ if (oldtup)
+ {
+ sz += sizeof(HeapTupleData);
+ oldlen = oldtup->tuple.t_len;
+ sz += oldlen;
+ }
+
+ if (newtup)
+ {
+ sz += sizeof(HeapTupleData);
+ newlen = newtup->tuple.t_len;
+ sz += newlen;
+ }
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ {
+ Size prefix_size = strlen(change->data.msg.prefix) + 1;
+
+ sz += prefix_size + change->data.msg.message_size +
+ sizeof(Size) + sizeof(Size);
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ {
+ Snapshot snap;
+
+ snap = change->data.snapshot;
+
+ sz += sizeof(SnapshotData) +
+ sizeof(TransactionId) * snap->xcnt +
+ sizeof(TransactionId) * snap->subxcnt;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ sz += sizeof(Oid) * change->data.truncate.nrelids;
+
+ break;
+ }
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ /* ReorderBufferChange contains everything important */
+ break;
+ }
+
+ return sz;
+}
+
+
+/*
* Restore a number of changes spilled to disk back into memory.
*/
static Size
@@ -2784,6 +3019,16 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
dlist_push_tail(&txn->changes, &change->node);
txn->nentries_mem++;
+
+ /*
+ * Update memory accounting for the restored change. We need to do this
+ * although we don't check the memory limit when restoring the changes in
+ * this branch (we only do that when initially queueing the changes after
+ * decoding), because we will release the changes later, and that will
+ * update the accounting too (subtracting the size from the counters).
+ * And we don't want to underflow there.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
@@ -3003,6 +3248,19 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn,
*
* We cannot replace unchanged toast tuples though, so those will still point
* to on-disk toast data.
+ *
+ * While updating the existing change with detoasted tuple data, we need to
+ * update the memory accounting info, because the change size will differ.
+ * Otherwise the accounting may get out of sync, triggering serialization
+ * at unexpected times.
+ *
+ * We simply subtract size of the change before rejiggering the tuple, and
+ * then adding the new size. This makes it look like the change was removed
+ * and then added back, except it only tweaks the accounting info.
+ *
+ * In particular it can't trigger serialization, which would be pointless
+ * anyway as it happens during commit processing right before handing
+ * the change to the output plugin.
*/
static void
ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
@@ -3023,6 +3281,13 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (txn->toast_hash == NULL)
return;
+ /*
+ * We're going modify the size of the change, so to make sure the
+ * accounting is correct we'll make it look like we're removing the
+ * change now (with the old size), and then re-add it at the end.
+ */
+ ReorderBufferChangeMemoryUpdate(rb, change, false);
+
oldcontext = MemoryContextSwitchTo(rb->context);
/* we should only have toast tuples in an INSERT or UPDATE */
@@ -3172,6 +3437,9 @@ ReorderBufferToastReplace(ReorderBuffer *rb, ReorderBufferTXN *txn,
pfree(isnull);
MemoryContextSwitchTo(oldcontext);
+
+ /* now add the change back, with the correct size */
+ ReorderBufferChangeMemoryUpdate(rb, change, true);
}
/*
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 11e6331..f737afb 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1725,6 +1725,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 9c08757..317c5d4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -21,6 +21,7 @@
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/inval.h"
#include "utils/int8.h"
#include "utils/memutils.h"
@@ -90,11 +91,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -140,6 +142,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,7 +199,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 2178e1c..5d7e687 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -65,6 +65,7 @@
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
#include "postmaster/walwriter.h"
+#include "replication/logical.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
#include "replication/syncrep.h"
@@ -191,6 +192,7 @@ static bool check_maxconnections(int *newval, void **extra, GucSource source);
static bool check_max_worker_processes(int *newval, void **extra, GucSource source);
static bool check_autovacuum_max_workers(int *newval, void **extra, GucSource source);
static bool check_max_wal_senders(int *newval, void **extra, GucSource source);
+static bool check_logical_decoding_work_mem(int *newval, void **extra, GucSource source);
static bool check_autovacuum_work_mem(int *newval, void **extra, GucSource source);
static bool check_effective_io_concurrency(int *newval, void **extra, GucSource source);
static void assign_effective_io_concurrency(int newval, void *extra);
@@ -2251,6 +2253,18 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"logical_decoding_work_mem", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum memory to be used for logical decoding."),
+ gettext_noop("This much memory can be used by each internal "
+ "reorder buffer before spilling to disk or streaming."),
+ GUC_UNIT_KB
+ },
+ &logical_decoding_work_mem,
+ -1, -1, MAX_KILOBYTES,
+ check_logical_decoding_work_mem, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
@@ -11286,6 +11300,28 @@ check_max_wal_senders(int *newval, void **extra, GucSource source)
}
static bool
+check_logical_decoding_work_mem(int *newval, void **extra, GucSource source)
+{
+ /*
+ * -1 indicates fallback.
+ *
+ * If we haven't yet changed the boot_val default of -1, just let it be.
+ * logical decoding will look to maintenance_work_mem instead.
+ */
+ if (*newval == -1)
+ return true;
+
+ /*
+ * We clamp manually-set values to at least 64kB. The maintenance_work_mem
+ * uses a higher minimum value (1MB), so this is OK.
+ */
+ if (*newval < 64)
+ *newval = 64;
+
+ return true;
+}
+
+static bool
check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
{
/*
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 0fc23e3..00a22b8 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -130,6 +130,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#logical_decoding_work_mem = 64MB # min 1MB, or -1 to use maintenance_work_mem
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c06a78..4dcef80 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -17,6 +17,8 @@
#include "utils/snapshot.h"
#include "utils/timestamp.h"
+extern PGDLLIMPORT int logical_decoding_work_mem;
+
/* an individual tuple, stored in one chunk of memory */
typedef struct ReorderBufferTupleBuf
{
@@ -63,6 +65,9 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_TRUNCATE
};
+/* forward declaration */
+struct ReorderBufferTXN;
+
/*
* a single 'change', can be an insert (with one tuple), an update (old, new),
* or a delete (old).
@@ -77,6 +82,9 @@ typedef struct ReorderBufferChange
/* The type of change. */
enum ReorderBufferChangeType action;
+ /* Transaction this change belongs to. */
+ struct ReorderBufferTXN *txn;
+
RepOriginId origin_id;
/*
@@ -286,6 +294,11 @@ typedef struct ReorderBufferTXN
*/
dlist_node node;
+ /*
+ * Size of this transaction (changes currently in memory, in bytes).
+ */
+ Size size;
+
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -386,6 +399,9 @@ struct ReorderBuffer
/* buffer for disk<->memory conversions */
char *outbuf;
Size outbufsize;
+
+ /* memory accounting */
+ Size size;
};
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934..4e68a69 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
0002-Immediately-WAL-log-assignments.patch (application/octet-stream, 19.1K) download
From f70b1e79f8dcb23f1acffd5a29758e7952e61d9c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:07:31 +0200
Subject: [PATCH 02/13] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So instead we write the assignment info into WAL immediately, as
part of the next WAL record (to minimize overhead).
---
src/backend/access/rmgrdesc/xactdesc.c | 26 ------
src/backend/access/transam/xact.c | 152 +++++++++----------------------
src/backend/access/transam/xlog.c | 2 -
src/backend/access/transam/xloginsert.c | 22 ++++-
src/backend/access/transam/xlogreader.c | 5 +
src/backend/replication/logical/decode.c | 39 ++++----
src/include/access/xact.h | 15 +--
src/include/access/xlog.h | 2 +
src/include/access/xlogreader.h | 3 +
src/include/access/xlogrecord.h | 1 +
src/tools/pgindent/typedefs.list | 1 -
11 files changed, 96 insertions(+), 172 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index a61f38d..66fc8fb 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -293,17 +293,6 @@ xact_desc_abort(StringInfo buf, uint8 info, xl_xact_abort *xlrec)
}
}
-static void
-xact_desc_assignment(StringInfo buf, xl_xact_assignment *xlrec)
-{
- int i;
-
- appendStringInfoString(buf, "subxacts:");
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- appendStringInfo(buf, " %u", xlrec->xsub[i]);
-}
-
void
xact_desc(StringInfo buf, XLogReaderState *record)
{
@@ -323,18 +312,6 @@ xact_desc(StringInfo buf, XLogReaderState *record)
xact_desc_abort(buf, XLogRecGetInfo(record), xlrec);
}
- else if (info == XLOG_XACT_ASSIGNMENT)
- {
- xl_xact_assignment *xlrec = (xl_xact_assignment *) rec;
-
- /*
- * Note that we ignore the WAL record's xid, since we're more
- * interested in the top-level xid that issued the record and which
- * xids are being reported here.
- */
- appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
- xact_desc_assignment(buf, xlrec);
- }
}
const char *
@@ -359,9 +336,6 @@ xact_identify(uint8 info)
case XLOG_XACT_ABORT_PREPARED:
id = "ABORT_PREPARED";
break;
- case XLOG_XACT_ASSIGNMENT:
- id = "ASSIGNMENT";
- break;
}
return id;
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 9162286..33141fb 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -188,9 +188,9 @@ typedef struct TransactionStateData
int prevSecContext; /* previous SecurityRestrictionContext */
bool prevXactReadOnly; /* entry-time xact r/o state */
bool startedInRecovery; /* did we start in recovery? */
- bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -225,13 +225,6 @@ static TransactionStateData TopTransactionStateData = {
.blockState = TBLOCK_DEFAULT,
};
-/*
- * unreportedXids holds XIDs of all subtransactions that have not yet been
- * reported in an XLOG_XACT_ASSIGNMENT record.
- */
-static int nUnreportedXids;
-static TransactionId unreportedXids[PGPROC_MAX_CACHED_SUBXIDS];
-
static TransactionState CurrentTransactionState = &TopTransactionStateData;
/*
@@ -502,19 +495,6 @@ GetCurrentFullTransactionIdIfAny(void)
}
/*
- * MarkCurrentTransactionIdLoggedIfAny
- *
- * Remember that the current xid - if it is assigned - now has been wal logged.
- */
-void
-MarkCurrentTransactionIdLoggedIfAny(void)
-{
- if (FullTransactionIdIsValid(CurrentTransactionState->fullTransactionId))
- CurrentTransactionState->didLogXid = true;
-}
-
-
-/*
* GetStableLatestTransactionId
*
* Get the transaction's XID if it has one, else read the next-to-be-assigned
@@ -555,7 +535,6 @@ AssignTransactionId(TransactionState s)
{
bool isSubXact = (s->parent != NULL);
ResourceOwner currentOwner;
- bool log_unknown_top = false;
/* Assert that caller didn't screw up */
Assert(!FullTransactionIdIsValid(s->fullTransactionId));
@@ -598,20 +577,6 @@ AssignTransactionId(TransactionState s)
}
/*
- * When wal_level=logical, guarantee that a subtransaction's xid can only
- * be seen in the WAL stream if its toplevel xid has been logged before.
- * If necessary we log an xact_assignment record with fewer than
- * PGPROC_MAX_CACHED_SUBXIDS. Note that it is fine if didLogXid isn't set
- * for a transaction even though it appears in a WAL record, we just might
- * superfluously log something. That can happen when an xid is included
- * somewhere inside a wal record, but not in XLogRecord->xl_xid, like in
- * xl_standby_locks.
- */
- if (isSubXact && XLogLogicalInfoActive() &&
- !TopTransactionStateData.didLogXid)
- log_unknown_top = true;
-
- /*
* Generate a new FullTransactionId and record its xid in PG_PROC and
* pg_subtrans.
*
@@ -646,59 +611,6 @@ AssignTransactionId(TransactionState s)
XactLockTableInsert(XidFromFullTransactionId(s->fullTransactionId));
CurrentResourceOwner = currentOwner;
-
- /*
- * Every PGPROC_MAX_CACHED_SUBXIDS assigned transaction ids within each
- * top-level transaction we issue a WAL record for the assignment. We
- * include the top-level xid and all the subxids that have not yet been
- * reported using XLOG_XACT_ASSIGNMENT records.
- *
- * This is required to limit the amount of shared memory required in a hot
- * standby server to keep track of in-progress XIDs. See notes for
- * RecordKnownAssignedTransactionIds().
- *
- * We don't keep track of the immediate parent of each subxid, only the
- * top-level transaction that each subxact belongs to. This is correct in
- * recovery only because aborted subtransactions are separately WAL
- * logged.
- *
- * This is correct even for the case where several levels above us didn't
- * have an xid assigned as we recursed up to them beforehand.
- */
- if (isSubXact && XLogStandbyInfoActive())
- {
- unreportedXids[nUnreportedXids] = XidFromFullTransactionId(s->fullTransactionId);
- nUnreportedXids++;
-
- /*
- * ensure this test matches similar one in
- * RecoverPreparedTransactions()
- */
- if (nUnreportedXids >= PGPROC_MAX_CACHED_SUBXIDS ||
- log_unknown_top)
- {
- xl_xact_assignment xlrec;
-
- /*
- * xtop is always set by now because we recurse up transaction
- * stack to the highest unassigned xid and then come back down
- */
- xlrec.xtop = GetTopTransactionId();
- Assert(TransactionIdIsValid(xlrec.xtop));
- xlrec.nsubxacts = nUnreportedXids;
-
- XLogBeginInsert();
- XLogRegisterData((char *) &xlrec, MinSizeOfXactAssignment);
- XLogRegisterData((char *) unreportedXids,
- nUnreportedXids * sizeof(TransactionId));
-
- (void) XLogInsert(RM_XACT_ID, XLOG_XACT_ASSIGNMENT);
-
- nUnreportedXids = 0;
- /* mark top, not current xact as having been logged */
- TopTransactionStateData.didLogXid = true;
- }
- }
}
/*
@@ -1792,13 +1704,6 @@ AtSubAbort_childXids(void)
s->childXids = NULL;
s->nChildXids = 0;
s->maxChildXids = 0;
-
- /*
- * We could prune the unreportedXids array here. But we don't bother. That
- * would potentially reduce number of XLOG_XACT_ASSIGNMENT records but it
- * would likely introduce more CPU time into the more common paths, so we
- * choose not to do that.
- */
}
/* ----------------------------------------------------------------
@@ -1963,12 +1868,6 @@ StartTransaction(void)
currentCommandIdUsed = false;
/*
- * initialize reported xid accounting
- */
- nUnreportedXids = 0;
- s->didLogXid = false;
-
- /*
* must initialize resource-management stuff first
*/
AtStart_Memory();
@@ -5095,6 +4994,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -5990,14 +5890,46 @@ xact_redo(XLogReaderState *record)
XLogRecGetOrigin(record));
LWLockRelease(TwoPhaseStateLock);
}
- else if (info == XLOG_XACT_ASSIGNMENT)
- {
- xl_xact_assignment *xlrec = (xl_xact_assignment *) XLogRecGetData(record);
-
- if (standbyState >= STANDBY_INITIALIZED)
- ProcArrayApplyXidAssignment(xlrec->xtop,
- xlrec->nsubxacts, xlrec->xsub);
- }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 790e2c8..b1daa05 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -1121,8 +1121,6 @@ XLogInsertRecord(XLogRecData *rdata,
*/
WALInsertLockRelease();
- MarkCurrentTransactionIdLoggedIfAny();
-
END_CRIT_SECTION();
/*
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 3ec67d4..15ce79c 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index c8b0d23..3b02fbf 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1072,6 +1072,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1110,6 +1111,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index c53e7e2..ff74c65 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -96,12 +96,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -220,12 +236,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -266,23 +282,6 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
DecodeAbort(ctx, buf, &parsed, xid);
break;
}
- case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index d714551..7553f84 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -145,7 +145,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_ABORT 0x20
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
-#define XLOG_XACT_ASSIGNMENT 0x50
+/* free opcode 0x50 */
/* free opcode 0x60 */
/* free opcode 0x70 */
@@ -188,15 +188,6 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XactCompletionForceSyncCommit(xinfo) \
((xinfo & XACT_COMPLETION_FORCE_SYNC_COMMIT) != 0)
-typedef struct xl_xact_assignment
-{
- TransactionId xtop; /* assigned XID's top-level XID */
- int nsubxacts; /* number of subtransaction XIDs */
- TransactionId xsub[FLEXIBLE_ARRAY_MEMBER]; /* assigned subxids */
-} xl_xact_assignment;
-
-#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
-
/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
@@ -363,7 +354,6 @@ extern FullTransactionId GetTopFullTransactionId(void);
extern FullTransactionId GetTopFullTransactionIdIfAny(void);
extern FullTransactionId GetCurrentFullTransactionId(void);
extern FullTransactionId GetCurrentFullTransactionIdIfAny(void);
-extern void MarkCurrentTransactionIdLoggedIfAny(void);
extern bool SubTransactionIsActive(SubTransactionId subxid);
extern CommandId GetCurrentCommandId(bool used);
extern void SetParallelStartTimestamps(TimestampTz xact_ts, TimestampTz stmt_ts);
@@ -410,6 +400,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252..060901d 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,8 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
+#define XLOG_INCLUDE_INVALS 0x08 /* include invalidations */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 1bbee38..c37a83d 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -148,6 +148,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -243,6 +245,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54..bcfba0a 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 60c76cb..d08c08a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3423,7 +3423,6 @@ xl_standby_locks
xl_tblspc_create_rec
xl_tblspc_drop_rec
xl_xact_abort
-xl_xact_assignment
xl_xact_commit
xl_xact_dbinfo
xl_xact_invals
--
1.8.3.1
0003-Issue-individual-invalidations-with-wal_level-logica.patch (application/octet-stream, 16.3K) download
From 2a2ac4be202b9226a12934ef7097764d6d8cb638 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:20:53 +0200
Subject: [PATCH 03/13] Issue individual invalidations with wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 52 ++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 17 ++++++
src/backend/replication/logical/reorderbuffer.c | 52 +++++++++++++++-
src/backend/utils/cache/inval.c | 81 +++++++++++++++++++++++++
src/include/access/xact.h | 18 +++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 238 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 66fc8fb..9cff9f0 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -312,6 +317,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
xact_desc_abort(buf, XLogRecGetInfo(record), xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -336,7 +349,46 @@ xact_identify(uint8 info)
case XLOG_XACT_ABORT_PREPARED:
id = "ABORT_PREPARED";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 33141fb..dc3633c 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -5890,6 +5890,13 @@ xact_redo(XLogReaderState *record)
XLogRecGetOrigin(record));
LWLockRelease(TwoPhaseStateLock);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index ff74c65..c100054 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -282,6 +282,23 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
DecodeAbort(ctx, buf, &parsed, xid);
break;
}
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 6228140..08b4d4f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -460,6 +460,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1811,6 +1812,18 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2204,6 +2217,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2643,6 +2688,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2739,6 +2785,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3014,6 +3061,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -3025,8 +3073,8 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
* although we don't check the memory limit when restoring the changes in
* this branch (we only do that when initially queueing the changes after
* decoding), because we will release the changes later, and that will
- * update the accounting too (subtracting the size from the counters).
- * And we don't want to underflow there.
+ * update the accounting too (subtracting the size from the counters). And
+ * we don't want to underflow there.
*/
ReorderBufferChangeMemoryUpdate(rb, change, true);
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9..f921fdf 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -543,6 +588,18 @@ RegisterSnapshotInvalidation(Oid dbId, Oid relId)
{
AddSnapshotInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.sn.id = SHAREDINVALSNAPSHOT_ID;
+ msg.sn.dbId = dbId;
+ msg.sn.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -1501,3 +1558,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7553f84..b26d399 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -145,7 +145,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_ABORT 0x20
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
-/* free opcode 0x50 */
+#define XLOG_XACT_INVALIDATIONS 0x50
/* free opcode 0x60 */
/* free opcode 0x70 */
@@ -189,6 +189,22 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
((xinfo & XACT_COMPLETION_FORCE_SYNC_COMMIT) != 0)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4dcef80..82dcb7f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -437,6 +448,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structure.patch (application/octet-stream, 7.9K) download
From bab2c69894e1a22bfe9a96f452644065150bd4a9 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH 05/13] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 34 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++----------
2 files changed, 37 insertions(+), 30 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 08b4d4f..ca4b904 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -728,7 +728,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -748,7 +748,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -771,7 +771,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -831,7 +831,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -847,7 +847,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1057,7 +1057,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1086,7 +1086,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1252,7 +1252,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1300,7 +1300,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1315,7 +1315,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1332,7 +1332,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1972,7 +1972,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2120,7 +2120,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2298,7 +2298,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2315,7 +2315,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2335,7 +2335,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2520,7 +2520,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e6d17fb..afe690a 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
0004-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From 0db9e31ccc2537fc3c37edea1280210644905dc1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH 04/13] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4b..9c77791 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..fc4ad65 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting. At
+ that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index da265f5..3230e45 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -69,6 +69,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -193,6 +208,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -867,6 +915,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 31c796b..d95d1b9 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -81,6 +81,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f..a305462 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 82dcb7f..e6d17fb 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
0006-Gracefully-handle-concurrent-aborts-of-uncommitted.patch (application/octet-stream, 13.7K) download
From 9777828b8f52fa1175fc1bf4ca2341b0bdb2cc9c Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 3 Oct 2019 09:00:49 +0530
Subject: [PATCH 06/13] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 32 +++++++++++++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 142 insertions(+), 9 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65..da6a6f3 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index e954482..6ce7878 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1433,16 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1714,16 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index ca4b904..1955bc5 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -679,7 +679,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1496,6 +1496,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
volatile CommandId command_id = FirstCommandId;
bool using_subtxn;
ReorderBufferIterTXNState *volatile iterstate = NULL;
+ MemoryContext ccxt = CurrentMemoryContext;
txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
false);
@@ -1529,7 +1530,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1780,7 +1781,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1800,7 +1801,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Every time the CommandId is incremented, we could
@@ -1879,6 +1880,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
+ /*
+ * if the catalog scan access returned an error of
+ * rollback, then abort on the other side as well
+ */
+ if (errdata->sqlerrcode == ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ elog(LOG, "stopping decoding of %s (%u)",
+ txn->gid[0] != '\0'? txn->gid:"", txn->xid);
+ rb->abort(rb, txn, commit_lsn);
+ }
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1902,7 +1917,14 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* remove potential on-disk data, and deallocate */
ReorderBufferCleanupTXN(rb, txn);
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ FlushErrorState();
}
PG_END_TRY();
}
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
0007-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 44.2K) download
From 63de67f99e363f506502b490965e662e6374d56e Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 3 Oct 2019 09:02:37 +0530
Subject: [PATCH 07/13] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 1074 ++++++++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1111 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 537e681..76a105a 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 1955bc5..c2d4677 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -358,6 +404,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -755,6 +804,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -851,6 +927,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -974,7 +1053,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1002,6 +1081,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1016,6 +1098,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1231,6 +1316,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
+/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1323,33 +1612,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1399,6 +1759,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1472,6 +1842,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1512,6 +1895,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_lsn = origin_lsn;
/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
* ReorderBufferCommitChild will have transferred any snapshots from
@@ -1546,6 +1945,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1562,6 +1962,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -2037,6 +2447,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2172,8 +2589,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2181,6 +2607,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2192,19 +2619,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2232,6 +2668,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->data.tuplecid.combocid = combocid;
change->lsn = lsn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2307,6 +2744,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2321,6 +2761,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2425,6 +2882,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2444,15 +2933,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- ReorderBufferSerializeTXN(rb, txn);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2737,6 +3257,504 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+
+ /*
+ * Every time the CommandId is incremented, we could
+ * see new catalog contents, so execute all
+ * invalidations.
+ */
+ ReorderBufferExecuteInvalidations(rb, txn);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index afe690a..f39b34f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
0008-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 89.6K) download
From 71e1f6727e3bedcd372d3995022c53d9eb011b23 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:56:18 +0200
Subject: [PATCH 08/13] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 7e3ba8e..a1c2a9f 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -72,6 +72,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index d85e831..5b3b095 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -67,7 +67,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -100,6 +101,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -194,6 +197,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -340,6 +363,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -356,7 +381,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -438,7 +464,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -705,11 +739,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -741,6 +778,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -753,7 +797,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -791,7 +835,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -828,7 +873,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 011076c..b22a053 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4106,6 +4106,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 65b3266..9baf61f 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 186057b..02574a0 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 3230e45..d5b5fe1 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1153,7 +1153,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1198,7 +1198,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47d..5a379fb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index f737afb..3493b02 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -59,6 +81,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -66,6 +89,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -105,6 +129,50 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
@@ -114,6 +182,9 @@ static void maybe_reread_subscription(void);
/* Flags set by signal handlers */
static volatile sig_atomic_t got_SIGHUP = false;
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -165,6 +236,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -512,6 +619,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -524,6 +943,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -539,6 +961,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -574,6 +999,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -677,6 +1105,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -797,6 +1228,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -896,6 +1330,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -987,6 +1424,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1084,6 +1537,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1097,6 +1566,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1546,6 +2018,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* SIGHUP: set flag to reload configuration at next convenient time */
static void
logicalrep_worker_sighup(SIGNAL_ARGS)
@@ -1726,6 +2756,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 317c5d4..0faa701 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -48,16 +48,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -67,6 +93,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -87,16 +114,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -165,6 +202,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -177,6 +231,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -200,7 +255,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -220,6 +276,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -287,9 +364,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -315,19 +425,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -336,6 +453,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -364,14 +485,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -381,7 +502,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -390,7 +511,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -416,6 +537,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -436,13 +561,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -516,6 +642,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -626,6 +837,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
+/*
* Relcache invalidation callback
*/
static void
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 42da631..507661c 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index eb4a98c..1c0c020 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -944,6 +944,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113..8793676 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fe076d8..bc45194 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -944,7 +944,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430a..bf02cbc 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 4e68a69..fe6acb4 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..4d01f7e
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..1a8b8ff
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..04af090
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..6fecfe6
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..50990c1
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
0010-Track-statistics-for-streaming-spilling.patch (application/octet-stream, 13.9K) download
From 5404d4ad25d52190c7297642014e695277aee0a1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:01:30 +0200
Subject: [PATCH 10/13] Track statistics for streaming/spilling
---
doc/src/sgml/monitoring.sgml | 47 +++++++++++++++++++
src/backend/catalog/system_views.sql | 8 +++-
src/backend/replication/logical/reorderbuffer.c | 21 +++++++++
src/backend/replication/walsender.c | 62 ++++++++++++++++++++++++-
src/include/catalog/pg_proc.dat | 6 +--
src/include/replication/reorderbuffer.h | 16 +++++++
src/include/replication/walsender_private.h | 10 ++++
src/test/regress/expected/rules.out | 10 +++-
8 files changed, 172 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 828e908..8de4fbd 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2121,6 +2121,53 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
with security-sensitive fields obfuscated.
</entry>
</row>
+ <row>
+ <entry><structfield>spill_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of transactions spilled to disk after the memory used by
+ logical decoding exceeds <literal>logical_work_mem</literal>. The
+ counter gets incremented both for toplevel transactions and
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>spill_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times transactions were spilled to disk. Transactions
+ may get spilled repeatedly, and this counter gets incremented on every
+ such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>spill_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded transaction data spilled to disk.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 9fe4a47..f8d0c4a 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -776,7 +776,13 @@ CREATE VIEW pg_stat_replication AS
W.replay_lag,
W.sync_priority,
W.sync_state,
- W.reply_time
+ W.reply_time,
+ W.spill_txns,
+ W.spill_count,
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 963cbf3..5e91522 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -354,6 +354,14 @@ ReorderBufferAllocate(void)
buffer->outbufsize = 0;
buffer->size = 0;
+ buffer->spillCount = 0;
+ buffer->spillTxns = 0;
+ buffer->spillBytes = 0;
+
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3001,6 +3009,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
int fd = -1;
XLogSegNo curOpenSegNo = 0;
Size spilled = 0;
+ Size size = txn->size;
elog(DEBUG2, "spill %u changes in XID %u to disk",
(uint32) txn->nentries_mem, txn->xid);
@@ -3059,6 +3068,11 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
spilled++;
}
+ /* update the statistics */
+ rb->spillCount += 1;
+ rb->spillTxns += (rbtxn_is_serialized(txn)) ? 1 : 0;
+ rb->spillBytes += size;
+
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
@@ -3763,6 +3777,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
PG_END_TRY();
/*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 1 : 0;
+ rb->streamBytes += txn->size;
+
+ /*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1c0c020..14abd85 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -248,6 +248,7 @@ static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
static bool TransactionIdInRecentPast(TransactionId xid, uint32 epoch);
+static void UpdateSpillStats(LogicalDecodingContext *ctx);
static void XLogRead(WALSegmentContext *segcxt, char *buf, XLogRecPtr startptr, Size count);
@@ -1267,7 +1268,8 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
/*
* LogicalDecodingContext 'update_progress' callback.
*
- * Write the current position to the lag tracker (see XLogSendPhysical).
+ * Write the current position to the lag tracker (see XLogSendPhysical),
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1286,6 +1288,12 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
LagTrackerWrite(lsn, now);
sendTime = now;
+
+ /*
+ * Update statistics about transactions that spilled to disk or
+ * streamed to subscriber (before being committed).
+ */
+ UpdateSpillStats(ctx);
}
/*
@@ -2325,6 +2333,12 @@ InitWalSenderSlot(void)
walsnd->state = WALSNDSTATE_STARTUP;
walsnd->latch = &MyProc->procLatch;
walsnd->replyTime = 0;
+ walsnd->spillTxns = 0;
+ walsnd->spillCount = 0;
+ walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3236,7 +3250,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 12
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3291,6 +3305,12 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int pid;
WalSndState state;
TimestampTz replyTime;
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3311,6 +3331,12 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
applyLag = walsnd->applyLag;
priority = walsnd->sync_standby_priority;
replyTime = walsnd->replyTime;
+ spillTxns = walsnd->spillTxns;
+ spillCount = walsnd->spillCount;
+ spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3392,6 +3418,16 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
nulls[11] = true;
else
values[11] = TimestampTzGetDatum(replyTime);
+
+ /* spill to disk */
+ values[12] = Int64GetDatum(spillTxns);
+ values[13] = Int64GetDatum(spillCount);
+ values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3628,3 +3664,25 @@ LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now)
Assert(time != 0);
return now - time;
}
+
+static void
+UpdateSpillStats(LogicalDecodingContext *ctx)
+{
+ ReorderBuffer *rb = ctx->reorder;
+
+ SpinLockAcquire(&MyWalSnd->mutex);
+
+ MyWalSnd->spillTxns = rb->spillTxns;
+ MyWalSnd->spillCount = rb->spillCount;
+ MyWalSnd->spillBytes = rb->spillBytes;
+
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(WARNING, "UpdateSpillStats: updating stats %p %ld %ld %ld %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes,
+ rb->streamTxns, rb->streamCount, rb->streamBytes);
+
+ SpinLockRelease(&MyWalSnd->mutex);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58ea5b9..9a508bf 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index f39b34f..53d9440 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -509,6 +509,22 @@ struct ReorderBuffer
/* memory accounting */
Size size;
+
+ /*
+ * Statistics about transactions streamed or spilled to disk.
+ *
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
+ */
+ int64 spillCount; /* spill-to-disk invocation counter */
+ int64 spillTxns; /* number of transactions spilled to disk */
+ int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 0dd6d1c..f726f25 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -80,6 +80,16 @@ typedef struct WalSnd
* Timestamp of the last message received from standby.
*/
TimestampTz replyTime;
+
+ /* Statistics for transactions spilled to disk. */
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 210e9cd..836abf0 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1951,9 +1951,15 @@ pg_stat_replication| SELECT s.pid,
w.replay_lag,
w.sync_priority,
w.sync_state,
- w.reply_time
+ w.reply_time,
+ w.spill_txns,
+ w.spill_count,
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
0009-Extend-the-concurrent-abort-handling-for-in-progress.patch (application/octet-stream, 2.1K) download
From 2a0712fb830a6be20296d18fb821862e302aeb8b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 3 Oct 2019 09:04:09 +0530
Subject: [PATCH 09/13] Extend the concurrent abort handling for in-progress
transaction
---
src/backend/replication/logical/reorderbuffer.c | 20 +++++++++++++++++++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c2d4677..963cbf3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -3280,6 +3280,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3410,6 +3411,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3722,6 +3730,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3740,7 +3751,14 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ FlushErrorState();
}
PG_END_TRY();
--
1.8.3.1
0011-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 16.8K) download
From 0bc62e1df19e19198d20330f54471abfe9618abd Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:16 +0200
Subject: [PATCH 11/13] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 4 ++--
src/test/subscription/t/010_stream_subxact.pl | 4 ++--
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 4 ++--
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 4 ++--
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 6 +++---
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 29 insertions(+), 29 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 40e306a..f41a0e1 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -64,7 +64,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f6..8dfeafc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 3ad00ea..78263c7 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR TABLE test_tab");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 4d01f7e..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -17,7 +17,7 @@ sub wait_for_caught_up
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
-$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
$node_publisher->start;
# Create subscriber node
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index 1a8b8ff..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -17,7 +17,7 @@ sub wait_for_caught_up
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
-$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
$node_publisher->start;
# Create subscriber node
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 04af090..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -17,7 +17,7 @@ sub wait_for_caught_up
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
-$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
$node_publisher->start;
# Create subscriber node
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 6fecfe6..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -17,7 +17,7 @@ sub wait_for_caught_up
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
-$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
$node_publisher->start;
# Create subscriber node
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index 50990c1..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -1,4 +1,4 @@
-# Test behavior with streaming transaction exceeding logical_work_mem
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
use strict;
use warnings;
use PostgresNode;
@@ -17,7 +17,7 @@ sub wait_for_caught_up
# Create publisher node
my $node_publisher = get_new_node('publisher');
$node_publisher->init(allows_streaming => 'logical');
-$node_publisher->append_conf('postgresql.conf', 'logical_work_mem = 64kB');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
$node_publisher->start;
# Create subscriber node
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
0012-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1010B) download
From 2fbbd9211397c5c79a25aa6b8160234d6fe4dfe6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH 12/13] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 5e91522..29da34d 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
0013-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From c3146318b04d5a6beb139c5c8120cdeee0a8a8b6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH 13/13] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v3-0014-BGWorkers-pool-for-streamed-transactions-apply.patch (application/octet-stream, 60.2K) download
From 9623e79695eb018f1e359e1a2b4b1be22a22d6a3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 3 Oct 2019 11:34:35 +0530
Subject: [PATCH] BGWorkers pool for streamed transactions apply
---
src/backend/postmaster/bgworker.c | 3 +
src/backend/postmaster/pgstat.c | 3 +
src/backend/replication/logical/proto.c | 17 +-
src/backend/replication/logical/worker.c | 1782 ++++++++++++++++--------------
src/include/pgstat.h | 1 +
src/include/replication/logicalproto.h | 6 +-
src/include/replication/logicalworker.h | 1 +
7 files changed, 936 insertions(+), 877 deletions(-)
diff --git a/src/backend/postmaster/bgworker.c b/src/backend/postmaster/bgworker.c
index b66b517..2e18083 100644
--- a/src/backend/postmaster/bgworker.c
+++ b/src/backend/postmaster/bgworker.c
@@ -129,6 +129,9 @@ static const struct
},
{
"ApplyWorkerMain", ApplyWorkerMain
+ },
+ {
+ "LogicalApplyBgwMain", LogicalApplyBgwMain
}
};
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index b22a053..46a75f2 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -3810,6 +3810,9 @@ pgstat_get_wait_ipc(WaitEventIPC w)
case WAIT_EVENT_HASH_GROW_BUCKETS_REINSERTING:
event_name = "Hash/GrowBuckets/Reinserting";
break;
+ case WAIT_EVENT_LOGICAL_APPLY_WORKER_READY:
+ event_name = "LogicalApplyWorkerReady";
+ break;
case WAIT_EVENT_LOGICAL_SYNC_DATA:
event_name = "LogicalSyncData";
break;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 5a379fb..d615b1f 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -788,14 +788,11 @@ logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
pq_sendint64(out, txn->commit_time);
}
-TransactionId
+void
logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
{
- TransactionId xid;
uint8 flags;
- xid = pq_getmsgint(in, 4);
-
/* read flags (unused for now) */
flags = pq_getmsgbyte(in);
@@ -806,8 +803,6 @@ logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
commit_data->commit_lsn = pq_getmsgint64(in);
commit_data->end_lsn = pq_getmsgint64(in);
commit_data->committime = pq_getmsgint64(in);
-
- return xid;
}
void
@@ -822,13 +817,3 @@ logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
pq_sendint32(out, xid);
pq_sendint32(out, subxid);
}
-
-void
-logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
- TransactionId *subxid)
-{
- Assert(xid && subxid);
-
- *xid = pq_getmsgint(in, 4);
- *subxid = pq_getmsgint(in, 4);
-}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 3493b02..7458ac0 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -81,11 +81,15 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/dsm.h"
#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "storage/shm_mq.h"
+#include "storage/shm_toc.h"
+#include "storage/spin.h"
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
@@ -101,6 +105,54 @@
#include "utils/timeout.h"
#define NAPTIME_PER_CYCLE 1000 /* max sleep time between cycles (1s) */
+#define PG_LOGICAL_APPLY_SHM_MAGIC 0x79fb2447 // TODO Consider change
+
+typedef struct ParallelState
+{
+ slock_t mutex;
+ // ConditionVariable cv;
+ bool attached;
+ bool ready;
+ bool finished;
+ Oid database_id;
+ Oid authenticated_user_id;
+ Oid subid;
+ Oid stream_xid;
+ uint32 n;
+} ParallelState;
+
+typedef struct WorkerState
+{
+ TransactionId xid;
+ BackgroundWorkerHandle *handle;
+ shm_mq_handle *mq_handle;
+ dsm_segment *dsm_seg;
+ ParallelState volatile *pstate;
+} WorkerState;
+
+/* Apply workers hash table (initialized on first use) */
+static HTAB *ApplyWorkersHash = NULL;
+static WorkerState **ApplyWorkersIdleList = NULL;
+static uint32 pool_size = 10; /* MaxConnections default? */
+static uint32 nworkers = 0;
+static uint32 nfreeworkers = 0;
+
+/* Fields valid only for apply background workers */
+bool isLogicalApplyWorker = false;
+volatile ParallelState *MyParallelState = NULL;
+
+/* Worker setup and interactions */
+static void setup_dsm(WorkerState *wstate);
+static void setup_background_worker(WorkerState *wstate);
+static void cleanup_background_worker(dsm_segment *seg, Datum arg);
+static void handle_sigterm(SIGNAL_ARGS);
+
+static bool check_worker_status(WorkerState *wstate);
+static void wait_for_worker(WorkerState *wstate);
+static void wait_for_worker_to_finish(WorkerState *wstate);
+
+static WorkerState * find_or_start_worker(TransactionId xid, bool start);
+static void stop_worker(WorkerState *wstate);
typedef struct FlushPosition
{
@@ -129,47 +181,13 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
-/* fields valid only when processing streamed transaction */
+/* Fields valid only when processing streamed transaction */
bool in_streamed_transaction = false;
static TransactionId stream_xid = InvalidTransactionId;
-static int stream_fd = -1;
-
-typedef struct SubXactInfo
-{
- TransactionId xid; /* XID of the subxact */
- off_t offset; /* offset in the file */
-} SubXactInfo;
-
-static uint32 nsubxacts = 0;
-static uint32 nsubxacts_max = 0;
-static SubXactInfo * subxacts = NULL;
-static TransactionId subxact_last = InvalidTransactionId;
-
-static void subxact_filename(char *path, Oid subid, TransactionId xid);
-static void changes_filename(char *path, Oid subid, TransactionId xid);
-
-/*
- * Information about subtransactions of a given toplevel transaction.
- */
-static void subxact_info_write(Oid subid, TransactionId xid);
-static void subxact_info_read(Oid subid, TransactionId xid);
-static void subxact_info_add(TransactionId xid);
-
-/*
- * Serialize and deserialize changes for a toplevel transaction.
- */
-static void stream_cleanup_files(Oid subid, TransactionId xid);
-static void stream_open_file(Oid subid, TransactionId xid, bool first);
-static void stream_write_change(char action, StringInfo s);
-static void stream_close_file(void);
-
-/*
- * Array of serialized XIDs.
- */
-static int nxids = 0;
-static int maxnxids = 0;
-static TransactionId *xids = NULL;
+static TransactionId current_xid = InvalidTransactionId;
+static TransactionId prev_xid = InvalidTransactionId;
+static uint32 nchanges = 0;
static bool handle_streamed_transaction(const char action, StringInfo s);
@@ -185,6 +203,16 @@ static volatile sig_atomic_t got_SIGHUP = false;
/* prototype needed because of stream_commit */
static void apply_dispatch(StringInfo s);
+// /* Debug only */
+// static void
+// iter_sleep(int seconds)
+// {
+// for (int i = 0; i < seconds; i++)
+// {
+// pg_usleep(1 * 1000L * 1000L);
+// }
+// }
+
/*
* Should this worker apply changes for given relation.
*
@@ -237,6 +265,107 @@ ensure_transaction(void)
}
/*
+ * Look up worker inside ApplyWorkersHash for requested xid.
+ * Throw error if not found or start a new one if start=true is passed.
+ */
+static WorkerState *
+find_or_start_worker(TransactionId xid, bool start)
+{
+ bool found;
+ WorkerState *entry = NULL;
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* First time through, initialize apply workers hashtable */
+ if (ApplyWorkersHash == NULL)
+ {
+ HASHCTL ctl;
+
+ MemSet(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(TransactionId);
+ ctl.entrysize = sizeof(WorkerState);
+ ctl.hcxt = ApplyContext; /* Allocate ApplyWorkersHash in the ApplyContext */
+ ApplyWorkersHash = hash_create("logical apply workers hash", 8,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ }
+
+ Assert(ApplyWorkersHash != NULL);
+
+ /*
+ * Find entry for requested transaction.
+ */
+ entry = hash_search(ApplyWorkersHash, &xid, HASH_FIND, &found);
+
+ if (!found && start)
+ {
+ /* If there is at least one worker in the idle list, then take one. */
+ if (nfreeworkers > 0)
+ {
+ char action = 'R';
+
+ Assert(ApplyWorkersIdleList != NULL);
+
+ entry = ApplyWorkersIdleList[nfreeworkers - 1];
+ if (!hash_update_hash_key(ApplyWorkersHash,
+ (void *) entry,
+ (void *) &xid))
+ elog(ERROR, "could not reassign apply worker #%u entry from xid %u to xid %u",
+ entry->pstate->n, entry->xid, xid);
+
+ entry->xid = xid;
+ entry->pstate->finished = false;
+ entry->pstate->stream_xid = xid;
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+
+ ApplyWorkersIdleList[--nfreeworkers] = NULL;
+ }
+ else
+ {
+ /* No entry in hash and no idle workers. Create a new one. */
+ entry = hash_search(ApplyWorkersHash, &xid, HASH_ENTER, &found);
+ entry->xid = xid;
+ setup_background_worker(entry);
+
+ if (nworkers == pool_size)
+ {
+ ApplyWorkersIdleList = repalloc(ApplyWorkersIdleList, pool_size + 10);
+ pool_size += 10;
+ }
+ }
+ }
+ else if (!found && !start)
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("could not find logical apply worker for xid %u", xid)));
+ else
+ elog(DEBUG5, "there is an existing logical apply worker for xid %u", xid);
+
+ Assert(entry != NULL);
+
+ return entry;
+}
+
+/*
+ * Gracefully teardown apply worker.
+ */
+static void
+stop_worker(WorkerState *wstate)
+{
+ /*
+ * Sending zero-length data to worker in order to stop it.
+ */
+ shm_mq_send(wstate->mq_handle, 0, NULL, false);
+
+ elog(LOG, "detaching DSM of apply worker #%u for xid %u",
+ wstate->pstate->n, wstate->xid);
+ dsm_detach(wstate->dsm_seg);
+
+ /* Delete worker entry */
+ (void) hash_search(ApplyWorkersHash, &wstate->xid, HASH_REMOVE, NULL);
+}
+
+/*
* Handle streamed transactions.
*
* If in streaming mode (receiving a block of streamed transaction), we
@@ -248,12 +377,12 @@ static bool
handle_streamed_transaction(const char action, StringInfo s)
{
TransactionId xid;
+ WorkerState *entry;
/* not in streaming mode */
- if (!in_streamed_transaction)
+ if (!in_streamed_transaction || isLogicalApplyWorker)
return false;
- Assert(stream_fd != -1);
Assert(TransactionIdIsValid(stream_xid));
/*
@@ -264,11 +393,16 @@ handle_streamed_transaction(const char action, StringInfo s)
Assert(TransactionIdIsValid(xid));
- /* Add the new subxact to the array (unless already there). */
- subxact_info_add(xid);
+ /*
+ * Find worker for requested xid.
+ */
+ entry = find_or_start_worker(stream_xid, false);
- /* write the change to the current file */
- stream_write_change(action, s);
+ // elog(LOG, "sending message of length=%d and raw=%s, action=%s", s->len, s->data, (char *) &action);
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
+ nchanges += 1;
+
+ // iter_sleep(3600);
return true;
}
@@ -624,7 +758,8 @@ apply_handle_origin(StringInfo s)
static void
apply_handle_stream_start(StringInfo s)
{
- bool first_segment;
+ bool first_segment;
+ WorkerState *entry;
Assert(!in_streamed_transaction);
@@ -633,17 +768,16 @@ apply_handle_stream_start(StringInfo s)
/* extract XID of the top-level transaction */
stream_xid = logicalrep_read_stream_start(s, &first_segment);
+ nchanges = 0;
- /* open the spool file for this transaction */
- stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, true);
- /*
- * if this is not the first segment, open existing file
- *
- * XXX Note that the cleanup is performed by stream_open_file.
- */
- if (!first_segment)
- subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+ SpinLockAcquire(&entry->pstate->mutex);
+ entry->pstate->ready = false;
+ SpinLockRelease(&entry->pstate->mutex);
+
+ elog(LOG, "starting streaming of xid %u", stream_xid);
pgstat_report_activity(STATE_RUNNING, NULL);
}
@@ -654,16 +788,19 @@ apply_handle_stream_start(StringInfo s)
static void
apply_handle_stream_stop(StringInfo s)
{
+ WorkerState *entry;
+ char action = 'E';
+
Assert(in_streamed_transaction);
- /*
- * Close the file with serialized changes, and serialize information about
- * subxacts for the toplevel transaction.
- */
- subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
- stream_close_file();
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, false);
+
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+ wait_for_worker(entry);
in_streamed_transaction = false;
+ elog(LOG, "stopped streaming of xid %u, %u changes streamed", stream_xid, nchanges);
pgstat_report_activity(STATE_IDLE, NULL);
}
@@ -676,96 +813,67 @@ apply_handle_stream_abort(StringInfo s)
{
TransactionId xid;
TransactionId subxid;
+ WorkerState *entry;
Assert(!in_streamed_transaction);
- logicalrep_read_stream_abort(s, &xid, &subxid);
-
- /*
- * If the two XIDs are the same, it's in fact abort of toplevel xact, so
- * just delete the files with serialized info.
- */
- if (xid == subxid)
+ if(isLogicalApplyWorker)
{
- char path[MAXPGPATH];
+ subxid = pq_getmsgint(s, 4);
- /*
- * XXX Maybe this should be an error instead? Can we receive abort for
- * a toplevel transaction we haven't received?
- */
+ ereport(LOG,
+ (errcode_for_file_access(),
+ errmsg("[Apply BGW #%u] aborting current transaction xid=%u, subxid=%u",
+ MyParallelState->n, GetCurrentTransactionIdIfAny(), GetCurrentSubTransactionId())));
- changes_filename(path, MyLogicalRepWorker->subid, xid);
+ if (subxid == stream_xid)
+ AbortCurrentTransaction();
+ else
+ {
+ char *spname = (char *) palloc(64 * sizeof(char));
+ sprintf(spname, "savepoint_for_xid_%u", subxid);
- if (unlink(path) < 0)
- ereport(ERROR,
+ ereport(LOG,
(errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
+ errmsg("[Apply BGW #%u] rolling back to savepoint %s", MyParallelState->n, spname)));
- subxact_filename(path, MyLogicalRepWorker->subid, xid);
-
- if (unlink(path) < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
+ RollbackToSavepoint(spname);
+ CommitTransactionCommand();
+ // RollbackAndReleaseCurrentSubTransaction();
- return;
+ pfree(spname);
+ }
}
else
{
- /*
- * OK, so it's a subxact. We need to read the subxact file for the
- * toplevel transaction, determine the offset tracked for the subxact,
- * and truncate the file with changes. We also remove the subxacts
- * with higher offsets (or rather higher XIDs).
- *
- * We intentionally scan the array from the tail, because we're likely
- * aborting a change for the most recent subtransactions.
- *
- * XXX Can we rely on the subxact XIDs arriving in sorted order? That
- * would allow us to use binary search here.
- *
- * XXX Or perhaps we can rely on the aborts to arrive in the reverse
- * order, i.e. from the inner-most subxact (when nested)? In which
- * case we could simply check the last element.
- */
+ xid = pq_getmsgint(s, 4);
+ subxid = pq_getmsgint(s, 4);
- int64 i;
- int64 subidx;
- bool found = false;
- char path[MAXPGPATH];
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(stream_xid, false);
- subidx = -1;
- subxact_info_read(MyLogicalRepWorker->subid, xid);
+ elog(LOG, "processing abort request of streamed transaction xid %u, subxid %u",
+ xid, subxid);
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
- /* FIXME optimize the search by bsearch on sorted data */
- for (i = nsubxacts; i > 0; i--)
+ if (subxid == stream_xid)
{
- if (subxacts[i - 1].xid == subxid)
- {
- subidx = (i - 1);
- found = true;
- break;
- }
- }
-
- /* We should not receive aborts for unknown subtransactions. */
- Assert(found);
+ char action = 'F';
+ shm_mq_send(entry->mq_handle, 1, &action, false);
+ // shm_mq_send(entry->mq_handle, 0, NULL, false);
- /* OK, truncate the file at the right offset. */
- Assert((subidx >= 0) && (subidx < nsubxacts));
+ wait_for_worker_to_finish(entry);
- changes_filename(path, MyLogicalRepWorker->subid, xid);
+ elog(LOG, "adding finished apply worker #%u for xid %u to the idle list",
+ entry->pstate->n, entry->xid);
+ ApplyWorkersIdleList[nfreeworkers++] = entry;
- if (truncate(path, subxacts[subidx].offset))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not truncate file \"%s\": %m", path)));
+ // elog(LOG, "detaching DSM of apply worker for xid=%u\n", entry->xid);
+ // dsm_detach(entry->dsm_seg);
- /* discard the subxacts added later */
- nsubxacts = subidx;
-
- /* write the updated subxact list */
- subxact_info_write(MyLogicalRepWorker->subid, xid);
+ // /* Delete worker entry */
+ // (void) hash_search(ApplyWorkersHash, &xid, HASH_REMOVE, NULL);
+ }
}
}
@@ -775,159 +883,56 @@ apply_handle_stream_abort(StringInfo s)
static void
apply_handle_stream_commit(StringInfo s)
{
- int fd;
TransactionId xid;
- StringInfoData s2;
- int nchanges;
-
- char path[MAXPGPATH];
- char *buffer = NULL;
+ WorkerState *entry;
LogicalRepCommitData commit_data;
- MemoryContext oldcxt;
-
- Assert(!in_streamed_transaction);
-
- xid = logicalrep_read_stream_commit(s, &commit_data);
-
- elog(DEBUG1, "received commit for streamed transaction %u", xid);
-
- /* open the spool file for the committed transaction */
- changes_filename(path, MyLogicalRepWorker->subid, xid);
-
- elog(DEBUG1, "replaying changes from file '%s'", path);
-
- fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
- if (fd < 0)
+ if (isLogicalApplyWorker)
{
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
- }
-
- /* XXX Should this be allocated in another memory context? */
+ // logicalrep_read_stream_commit(s, &commit_data);
- oldcxt = MemoryContextSwitchTo(TopMemoryContext);
-
- buffer = palloc(8192);
- initStringInfo(&s2);
-
- MemoryContextSwitchTo(oldcxt);
-
- ensure_transaction();
-
- /*
- * Make sure the handle apply_dispatch methods are aware we're in a remote
- * transaction.
- */
- in_remote_transaction = true;
- pgstat_report_activity(STATE_RUNNING, NULL);
-
- /*
- * Read the entries one by one and pass them through the same logic as in
- * apply_dispatch.
- */
- nchanges = 0;
- while (true)
+ CommitTransactionCommand();
+ }
+ else
{
- int nbytes;
- int len;
-
- /* read length of the on-disk record */
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
- nbytes = read(fd, &len, sizeof(len));
- pgstat_report_wait_end();
-
- /* have we reached end of the file? */
- if (nbytes == 0)
- break;
-
- /* do we have a correct length? */
- if (nbytes != sizeof(len))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file: %m")));
- return;
- }
-
- Assert(len > 0);
+ char action = 'F';
- /* make sure we have sufficiently large buffer */
- buffer = repalloc(buffer, len);
-
- /* and finally read the data into the buffer */
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
- if (read(fd, buffer, len) != len)
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file: %m")));
- return;
- }
- pgstat_report_wait_end();
+ Assert(!in_streamed_transaction);
- /* copy the buffer to the stringinfo and call apply_dispatch */
- resetStringInfo(&s2);
- appendBinaryStringInfo(&s2, buffer, len);
+ xid = pq_getmsgint(s, 4);
+ logicalrep_read_stream_commit(s, &commit_data);
- /* Ensure we are reading the data into our memory context. */
- oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
- apply_dispatch(&s2);
+ /* Find worker for requested xid */
+ entry = find_or_start_worker(xid, false);
- MemoryContextReset(ApplyMessageContext);
+ /* Send commit message */
+ shm_mq_send(entry->mq_handle, s->len, s->data, false);
- MemoryContextSwitchTo(oldcxt);
+ /* Notify worker, that we are done with this xact */
+ shm_mq_send(entry->mq_handle, 1, &action, false);
- nchanges++;
+ wait_for_worker_to_finish(entry);
- if (nchanges % 1000 == 0)
- elog(DEBUG1, "replayed %d changes from file '%s'",
- nchanges, path);
+ elog(LOG, "adding finished apply worker #%u for xid %u to the idle list",
+ entry->pstate->n, entry->xid);
+ ApplyWorkersIdleList[nfreeworkers++] = entry;
/*
- * send feedback to upstream
- *
- * XXX Probably should send a valid LSN. But which one?
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
*/
- send_feedback(InvalidXLogRecPtr, false, false);
- }
-
- CloseTransientFile(fd);
-
- /*
- * Update origin state so we can restart streaming from correct
- * position in case of crash.
- */
- replorigin_session_origin_lsn = commit_data.end_lsn;
- replorigin_session_origin_timestamp = commit_data.committime;
-
- CommitTransactionCommand();
- pgstat_report_stat(false);
-
- store_flush_position(commit_data.end_lsn);
-
- elog(DEBUG1, "replayed %d (all) changes from file '%s'",
- nchanges, path);
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
- in_remote_transaction = false;
- pgstat_report_activity(STATE_IDLE, NULL);
+ pgstat_report_stat(false);
- /* unlink the files with serialized changes and subxact info */
- stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+ store_flush_position(commit_data.end_lsn);
- pfree(buffer);
- pfree(s2.data);
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+ }
}
/*
@@ -946,6 +951,8 @@ apply_handle_relation(StringInfo s)
if (handle_streamed_transaction('R', s))
return;
+ // iter_sleep(3600);
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -1386,6 +1393,38 @@ apply_dispatch(StringInfo s)
{
char action = pq_getmsgbyte(s);
+ if (isLogicalApplyWorker)
+ {
+ /*
+ * Inside logical apply worker we can figure out that new subtransaction
+ * was started if new change arrived with different xid. In that case we
+ * can define named savepoint, so that we were able to commit/rollback it
+ * separately later.
+ *
+ * Special case is if the first change comes from subtransuction, then
+ * we check that current_xid differs from stream_xid.
+ */
+ current_xid = pq_getmsgint(s, 4);
+
+ if (current_xid != stream_xid
+ && ((TransactionIdIsValid(prev_xid) && current_xid != prev_xid)
+ || !TransactionIdIsValid(prev_xid)))
+ {
+ char *spname = (char *) palloc(64 * sizeof(char));
+ sprintf(spname, "savepoint_for_xid_%u", current_xid);
+
+ elog(LOG, "[Apply BGW #%u] defining savepoint %s", MyParallelState->n, spname);
+
+ DefineSavepoint(spname);
+ CommitTransactionCommand();
+ // BeginInternalSubTransaction(NULL);
+ }
+
+ prev_xid = current_xid;
+ }
+ // else
+ // elog(LOG, "Logical worker: applying dispatch for action=%s", (char *) &action);
+
switch (action)
{
/* BEGIN */
@@ -1414,6 +1453,7 @@ apply_dispatch(StringInfo s)
break;
/* RELATION */
case 'R':
+ // elog(LOG, "%s worker: applying dispatch for action=R", isLogicalApplyWorker ? "Apply" : "Logical");
apply_handle_relation(s);
break;
/* TYPE */
@@ -1544,12 +1584,18 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
static void
worker_onexit(int code, Datum arg)
{
- int i;
+ HASH_SEQ_STATUS status;
+ WorkerState *entry;
- elog(LOG, "cleanup files for %d transactions", nxids);
-
- for (i = nxids-1; i >= 0; i--)
- stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+ if (ApplyWorkersHash != NULL)
+ {
+ hash_seq_init(&status, ApplyWorkersHash);
+ while ((entry = (WorkerState *) hash_seq_search(&status)) != NULL)
+ {
+ stop_worker(entry);
+ }
+ hash_seq_term(&status);
+ }
}
/*
@@ -1572,6 +1618,8 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
+ ApplyWorkersIdleList = palloc(sizeof(WorkerState *) * pool_size);
+
for (;;)
{
pgsocket fd = PGINVALID_SOCKET;
@@ -1883,8 +1931,9 @@ maybe_reread_subscription(void)
Subscription *newsub;
bool started_tx = false;
+ // TODO Probably we have to handle subscription reread in apply workers too.
/* When cache state is valid there is nothing to do here. */
- if (MySubscriptionValid)
+ if (MySubscriptionValid || isLogicalApplyWorker)
return;
/* This function might be called inside or outside of transaction. */
@@ -2018,608 +2067,50 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
-/*
- * subxact_info_write
- * Store information about subxacts for a toplevel transaction.
- *
- * For each subxact we store offset of it's first change in the main file.
- * The file is always over-written as a whole, and we also include CRC32C
- * checksum of the information.
- *
- * XXX We should only store subxacts that were not aborted yet.
- *
- * XXX Maybe we should only include the checksum when the cluster is
- * initialized with checksums?
- *
- * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
- */
+/* SIGHUP: set flag to reload configuration at next convenient time */
static void
-subxact_info_write(Oid subid, TransactionId xid)
+logicalrep_worker_sighup(SIGNAL_ARGS)
{
- int fd;
- char path[MAXPGPATH];
- uint32 checksum;
- Size len;
-
- Assert(TransactionIdIsValid(xid));
-
- subxact_filename(path, subid, xid);
-
- fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
- if (fd < 0)
- {
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not create file \"%s\": %m",
- path)));
- return;
- }
-
- len = sizeof(SubXactInfo) * nsubxacts;
-
- /* compute the checksum */
- INIT_CRC32C(checksum);
- COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
- COMP_CRC32C(checksum, (char *) subxacts, len);
- FIN_CRC32C(checksum);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
-
- if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
- {
- int save_errno = errno;
+ int save_errno = errno;
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ got_SIGHUP = true;
- if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
- {
- int save_errno = errno;
+ /* Waken anything waiting on the process latch */
+ SetLatch(MyLatch);
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ errno = save_errno;
+}
- if ((len > 0) && (write(fd, subxacts, len) != len))
- {
- int save_errno = errno;
+/* Logical Replication Apply worker entry point */
+void
+ApplyWorkerMain(Datum main_arg)
+{
+ int worker_slot = DatumGetInt32(main_arg);
+ MemoryContext oldctx;
+ char originname[NAMEDATALEN];
+ XLogRecPtr origin_startpos;
+ char *myslotname;
+ WalRcvStreamOptions options;
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not write to file \"%s\": %m",
- path)));
- return;
- }
+ /* Attach to slot */
+ logicalrep_worker_attach(worker_slot);
- pgstat_report_wait_end();
+ /* Setup signal handling */
+ pqsignal(SIGHUP, logicalrep_worker_sighup);
+ pqsignal(SIGTERM, die);
+ BackgroundWorkerUnblockSignals();
/*
- * We don't need to fsync or anything, as we'll recreate the files after a
- * crash from scratch. So just close the file.
+ * We don't currently need any ResourceOwner in a walreceiver process, but
+ * if we did, we could call CreateAuxProcessResourceOwner here.
*/
- CloseTransientFile(fd);
- /*
- * But we free the memory allocated for subxact info. There might be one
- * exceptional transaction with many subxacts, and we don't want to keep
- * the memory allocated forewer.
- *
- */
- if (subxacts)
- pfree(subxacts);
+ /* Initialise stats to a sanish value */
+ MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
+ MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
- subxacts = NULL;
- subxact_last = InvalidTransactionId;
- nsubxacts = 0;
- nsubxacts_max = 0;
-}
-
-/*
- * subxact_info_read
- * Restore information about subxacts of a streamed transaction.
- *
- * Read information about subxacts into the global variables, and while
- * reading the information verify the checksum.
- *
- * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
- *
- * XXX Do we need to allocate it in TopMemoryContext?
- */
-static void
-subxact_info_read(Oid subid, TransactionId xid)
-{
- int fd;
- char path[MAXPGPATH];
- uint32 checksum;
- uint32 checksum_new;
- Size len;
- MemoryContext oldctx;
-
- Assert(TransactionIdIsValid(xid));
- Assert(!subxacts);
- Assert(nsubxacts == 0);
- Assert(nsubxacts_max == 0);
-
- subxact_filename(path, subid, xid);
-
- fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
- if (fd < 0)
- {
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
-
- /* read the checksum */
- if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- /* read number of subxact items */
- if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_end();
-
- len = sizeof(SubXactInfo) * nsubxacts;
-
- /* we keep the maximum as a power of 2 */
- nsubxacts_max = 1 << my_log2(nsubxacts);
-
- /* subxacts are long-lived */
- oldctx = MemoryContextSwitchTo(TopMemoryContext);
- subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
- MemoryContextSwitchTo(oldctx);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
-
- if ((len > 0) && ((read(fd, subxacts, len)) != len))
- {
- int save_errno = errno;
-
- CloseTransientFile(fd);
- errno = save_errno;
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not read file \"%s\": %m",
- path)));
- return;
- }
-
- pgstat_report_wait_end();
-
- /* recompute the checksum */
- INIT_CRC32C(checksum_new);
- COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
- COMP_CRC32C(checksum_new, (char *) subxacts, len);
- FIN_CRC32C(checksum_new);
-
- if (checksum_new != checksum)
- ereport(ERROR,
- (errmsg("checksum failure when reading subxacts")));
-
- CloseTransientFile(fd);
-}
-
-/*
- * subxact_info_add
- * Add information about a subxact (offset in the main file).
- *
- * XXX Do we need to allocate it in TopMemoryContext?
- */
-static void
-subxact_info_add(TransactionId xid)
-{
- int64 i;
-
- /*
- * If the XID matches the toplevel transaction, we don't want to add it.
- */
- if (stream_xid == xid)
- return;
-
- /*
- * In most cases we're checking the same subxact as we've already seen in
- * the last call, so make ure just ignore it (this change comes later).
- */
- if (subxact_last == xid)
- return;
-
- /* OK, remember we're processing this XID. */
- subxact_last = xid;
-
- /*
- * Check if the transaction is already present in the array of subxact. We
- * intentionally scan the array from the tail, because we're likely adding
- * a change for the most recent subtransactions.
- *
- * XXX Can we rely on the subxact XIDs arriving in sorted order? That
- * would allow us to use binary search here.
- */
- for (i = nsubxacts; i > 0; i--)
- {
- /* found, so we're done */
- if (subxacts[i - 1].xid == xid)
- return;
- }
-
- /* This is a new subxact, so we need to add it to the array. */
-
- if (nsubxacts == 0)
- {
- MemoryContext oldctx;
-
- nsubxacts_max = 128;
- oldctx = MemoryContextSwitchTo(TopMemoryContext);
- subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
- MemoryContextSwitchTo(oldctx);
- }
- else if (nsubxacts == nsubxacts_max)
- {
- nsubxacts_max *= 2;
- subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
- }
-
- subxacts[nsubxacts].xid = xid;
- subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
-
- nsubxacts++;
-}
-
-/* format filename for file containing the info about subxacts */
-static void
-subxact_filename(char *path, Oid subid, TransactionId xid)
-{
- char tempdirpath[MAXPGPATH];
-
- TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
-
- /*
- * We might need to create the tablespace's tempfile directory, if no
- * one has yet done so.
- *
- * Don't check for error from mkdir; it could fail if the directory
- * already exists (maybe someone else just did the same thing). If
- * it doesn't work then we'll bomb out when opening the file
- */
- mkdir(tempdirpath, S_IRWXU);
-
- snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
- tempdirpath, subid, xid);
-}
-
-/* format filename for file containing serialized changes */
-static void
-changes_filename(char *path, Oid subid, TransactionId xid)
-{
- char tempdirpath[MAXPGPATH];
-
- TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
-
- /*
- * We might need to create the tablespace's tempfile directory, if no
- * one has yet done so.
- *
- * Don't check for error from mkdir; it could fail if the directory
- * already exists (maybe someone else just did the same thing). If
- * it doesn't work then we'll bomb out when opening the file
- */
- mkdir(tempdirpath, S_IRWXU);
-
- snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
- tempdirpath, subid, xid);
-}
-
-/*
- * stream_cleanup_files
- * Cleanup files for a subscription / toplevel transaction.
- *
- * Remove files with serialized changes and subxact info for a particular
- * toplevel transaction. Each subscription has a separate set of files.
- *
- * Note: The files may not exists, so handle ENOENT as non-error.
- *
- * TODO: Add missing_ok flag to specify in which cases it's OK not to
- * find the files, and when it's an error.
- */
-static void
-stream_cleanup_files(Oid subid, TransactionId xid)
-{
- int i;
- char path[MAXPGPATH];
- bool found = false;
-
- subxact_filename(path, subid, xid);
-
- if ((unlink(path) < 0) && (errno != ENOENT))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
-
- changes_filename(path, subid, xid);
-
- if ((unlink(path) < 0) && (errno != ENOENT))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
-
- /*
- * Cleanup the XID from the array - find the XID in the array and
- * remove it by shifting all the remaining elements. The array is
- * bound to be fairly small (maximum number of in-progress xacts,
- * so max_connections + max_prepared_transactions) so simply loop
- * through the array and find index of the XID. Then move the rest
- * of the array by one element to the left.
- *
- * Notice we also call this from stream_open_file for first segment
- * of each transaction, to deal with possible left-overs after a
- * crash, so it's entirely possible not to find the XID in the
- * array here. In that case we don't remove anything.
- *
- * XXX Perhaps it'd be better to handle this automatically after a
- * restart, instead of doing it over and over for each transaction.
- */
- for (i = 0; i < nxids; i++)
- {
- if (xids[i] == xid)
- {
- found = true;
- break;
- }
- }
-
- if (!found)
- return;
-
- /*
- * Move the last entry from the array to the place. We don't keep
- * the streamed transactions sorted or anything - we only expect
- * a few of them in progress (max_connections + max_prepared_xacts)
- * so linear search is just fine.
- */
- xids[i] = xids[nxids-1];
- nxids--;
-}
-
-/*
- * stream_open_file
- * Open file we'll use to serialize changes for a toplevel transaction.
- *
- * Open a file for streamed changes from a toplevel transaction identified
- * by stream_xid (global variable). If it's the first chunk of streamed
- * changes for this transaction, perform cleanup by removing existing
- * files after a possible previous crash.
- *
- * This can only be called at the beginning of a "streaming" block, i.e.
- * between stream_start/stream_stop messages from the upstream.
- */
-static void
-stream_open_file(Oid subid, TransactionId xid, bool first_segment)
-{
- char path[MAXPGPATH];
- int flags;
-
- Assert(in_streamed_transaction);
- Assert(OidIsValid(subid));
- Assert(TransactionIdIsValid(xid));
- Assert(stream_fd == -1);
-
- /*
- * If this is the first segment for this transaction, try removing
- * existing files (if there are any, possibly after a crash).
- */
- if (first_segment)
- {
- MemoryContext oldcxt;
-
- /* XXX make sure there are no previous files for this transaction */
- stream_cleanup_files(subid, xid);
-
- oldcxt = MemoryContextSwitchTo(TopMemoryContext);
-
- /*
- * We need to remember the XIDs we spilled to files, so that we can
- * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
- *
- * The number of XIDs we may need to track is fairly small, because
- * we can only stream toplevel xacts (so limited by max_connections
- * and max_prepared_transactions), and we only stream the large ones.
- * So we simply keep the XIDs in an unsorted array. If the number of
- * xacts gets large for some reason (e.g. very high max_connections),
- * a more elaborate approach might be better - e.g. sorted array, to
- * speed-up the lookups.
- */
- if (nxids == maxnxids) /* array of XIDs is full */
- {
- if (!xids)
- {
- maxnxids = 64;
- xids = palloc(maxnxids * sizeof(TransactionId));
- }
- else
- {
- maxnxids = 2 * maxnxids;
- xids = repalloc(xids, maxnxids * sizeof(TransactionId));
- }
- }
-
- xids[nxids++] = xid;
-
- MemoryContextSwitchTo(oldcxt);
- }
-
- changes_filename(path, subid, xid);
-
- elog(DEBUG1, "opening file '%s' for streamed changes", path);
-
- /*
- * If this is the first streamed segment, the file must not exist, so
- * make sure we're the ones creating it. Otherwise just open the file
- * for writing, in append mode.
- */
- if (first_segment)
- flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
- else
- flags = (O_WRONLY | O_APPEND | PG_BINARY);
-
- stream_fd = OpenTransientFile(path, flags);
-
- if (stream_fd < 0)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not open file \"%s\": %m",
- path)));
-}
-
-/*
- * stream_close_file
- * Close the currently open file with streamed changes.
- *
- * This can only be called at the beginning of a "streaming" block, i.e.
- * between stream_start/stream_stop messages from the upstream.
- */
-static void
-stream_close_file(void)
-{
- Assert(in_streamed_transaction);
- Assert(TransactionIdIsValid(stream_xid));
- Assert(stream_fd != -1);
-
- CloseTransientFile(stream_fd);
-
- stream_xid = InvalidTransactionId;
- stream_fd = -1;
-}
-
-/*
- * stream_write_change
- * Serialize a change to a file for the current toplevel transaction.
- *
- * The change is serialied in a simple format, with length (not including
- * the length), action code (identifying the message type) and message
- * contents (without the subxact TransactionId value).
- *
- * XXX The subxact file includes CRC32C of the contents. Maybe we should
- * include something like that here too, but doing so will not be as
- * straighforward, because we write the file in chunks.
- */
-static void
-stream_write_change(char action, StringInfo s)
-{
- int len;
-
- Assert(in_streamed_transaction);
- Assert(TransactionIdIsValid(stream_xid));
- Assert(stream_fd != -1);
-
- /* total on-disk size, including the action type character */
- len = (s->len - s->cursor) + sizeof(char);
-
- pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
-
- /* first write the size */
- if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- /* then the action */
- if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- /* and finally the remaining part of the buffer (after the XID) */
- len = (s->len - s->cursor);
-
- if (write(stream_fd, &s->data[s->cursor], len) != len)
- ereport(ERROR,
- (errcode_for_file_access(),
- errmsg("could not serialize streamed change to file: %m")));
-
- pgstat_report_wait_end();
-}
-
-/* SIGHUP: set flag to reload configuration at next convenient time */
-static void
-logicalrep_worker_sighup(SIGNAL_ARGS)
-{
- int save_errno = errno;
-
- got_SIGHUP = true;
-
- /* Waken anything waiting on the process latch */
- SetLatch(MyLatch);
-
- errno = save_errno;
-}
-
-/* Logical Replication Apply worker entry point */
-void
-ApplyWorkerMain(Datum main_arg)
-{
- int worker_slot = DatumGetInt32(main_arg);
- MemoryContext oldctx;
- char originname[NAMEDATALEN];
- XLogRecPtr origin_startpos;
- char *myslotname;
- WalRcvStreamOptions options;
-
- /* Attach to slot */
- logicalrep_worker_attach(worker_slot);
-
- /* Setup signal handling */
- pqsignal(SIGHUP, logicalrep_worker_sighup);
- pqsignal(SIGTERM, die);
- BackgroundWorkerUnblockSignals();
-
- /*
- * We don't currently need any ResourceOwner in a walreceiver process, but
- * if we did, we could call CreateAuxProcessResourceOwner here.
- */
-
- /* Initialise stats to a sanish value */
- MyLogicalRepWorker->last_send_time = MyLogicalRepWorker->last_recv_time =
- MyLogicalRepWorker->reply_time = GetCurrentTimestamp();
-
- /* Load the libpq-specific functions */
- load_file("libpqwalreceiver", false);
+ /* Load the libpq-specific functions */
+ load_file("libpqwalreceiver", false);
/* Run as replica session replication role. */
SetConfigOption("session_replication_role", "replica",
@@ -2775,3 +2266,580 @@ IsLogicalWorker(void)
{
return MyLogicalRepWorker != NULL;
}
+
+/*
+ * Apply Background Worker main loop.
+ */
+void
+LogicalApplyBgwMain(Datum main_arg)
+{
+ volatile ParallelState *pst;
+
+ dsm_segment *seg;
+ shm_toc *toc;
+ PGPROC *registrant;
+ shm_mq *mq;
+ shm_mq_handle *mqh;
+ shm_mq_result shmq_res;
+ // ConditionVariable cv;
+ LogicalRepWorker lrw;
+ MemoryContext oldcontext;
+
+ MemoryContextSwitchTo(TopMemoryContext);
+
+ /* Load the subscription into persistent memory context. */
+ ApplyContext = AllocSetContextCreate(TopMemoryContext,
+ "ApplyContext",
+ ALLOCSET_DEFAULT_SIZES);
+
+ oldcontext = MemoryContextSwitchTo(ApplyContext);
+
+ /*
+ * Init the ApplyMessageContext which we clean up after each replication
+ * protocol message.
+ */
+ ApplyMessageContext = AllocSetContextCreate(ApplyContext,
+ "ApplyMessageContext",
+ ALLOCSET_DEFAULT_SIZES);
+
+ isLogicalApplyWorker = true;
+
+ /*
+ * Establish signal handlers.
+ *
+ * We want CHECK_FOR_INTERRUPTS() to kill off this worker process just as
+ * it would a normal user backend. To make that happen, we establish a
+ * signal handler that is a stripped-down version of die().
+ */
+ pqsignal(SIGTERM, handle_sigterm);
+ BackgroundWorkerUnblockSignals();
+
+ /*
+ * Connect to the dynamic shared memory segment.
+ *
+ * The backend that registered this worker passed us the ID of a shared
+ * memory segment to which we must attach for further instructions. In
+ * order to attach to dynamic shared memory, we need a resource owner.
+ * Once we've mapped the segment in our address space, attach to the table
+ * of contents so we can locate the various data structures we'll need to
+ * find within the segment.
+ */
+ CurrentResourceOwner = ResourceOwnerCreate(NULL, "Logical apply worker");
+ seg = dsm_attach(DatumGetInt32(main_arg));
+ if (seg == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("unable to map dynamic shared memory segment")));
+ toc = shm_toc_attach(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg));
+ if (toc == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("bad magic number in dynamic shared memory segment")));
+
+ /*
+ * Acquire a worker number.
+ *
+ * By convention, the process registering this background worker should
+ * have stored the control structure at key 0. We look up that key to
+ * find it. Our worker number gives our identity: there may be just one
+ * worker involved in this parallel operation, or there may be many.
+ */
+ pst = shm_toc_lookup(toc, 0, false);
+ MyParallelState = pst;
+
+ SpinLockAcquire(&pst->mutex);
+ pst->attached = true;
+ SpinLockRelease(&pst->mutex);
+
+ /*
+ * Attach to the message queue.
+ */
+ mq = shm_toc_lookup(toc, 1, false);
+ shm_mq_set_receiver(mq, MyProc);
+ mqh = shm_mq_attach(mq, seg, NULL);
+
+ /* Restore database connection. */
+ BackgroundWorkerInitializeConnectionByOid(pst->database_id,
+ pst->authenticated_user_id, 0);
+
+ /*
+ * Set the client encoding to the database encoding, since that is what
+ * the leader will expect.
+ */
+ SetClientEncoding(GetDatabaseEncoding());
+
+ lrw.subid = pst->subid;
+ MyLogicalRepWorker = &lrw;
+
+ stream_xid = pst->stream_xid;
+
+ StartTransactionCommand();
+ BeginTransactionBlock();
+ CommitTransactionCommand();
+ StartTransactionCommand();
+ // PushActiveSnapshot(GetTransactionSnapshot());
+
+ MySubscription = GetSubscription(MyLogicalRepWorker->subid, true);
+
+ /*
+ * Indicate that we're fully initialized and ready to begin the main part
+ * of the parallel operation.
+ *
+ * Once we signal that we're ready, the user backend is entitled to assume
+ * that our on_dsm_detach callbacks will fire before we disconnect from
+ * the shared memory segment and exit. Generally, that means we must have
+ * attached to all relevant dynamic shared memory data structures by now.
+ */
+ SpinLockAcquire(&pst->mutex);
+ pst->ready = true;
+ // cv = pst->cv;
+ // if (pst->workers_ready == pst->workers_total)
+ // {
+ // registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ // if (registrant == NULL)
+ // {
+ // elog(DEBUG1, "registrant backend has exited prematurely");
+ // proc_exit(1);
+ // }
+ // SetLatch(®istrant->procLatch);
+ // }
+ SpinLockRelease(&pst->mutex);
+ elog(LOG, "[Apply BGW #%u] started", pst->n);
+
+ registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ SetLatch(®istrant->procLatch);
+
+ for (;;)
+ {
+ void *data;
+ Size len;
+ StringInfoData s;
+ MemoryContext oldctx;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* Ensure we are reading the data into our memory context. */
+ oldctx = MemoryContextSwitchTo(ApplyMessageContext);
+
+ shmq_res = shm_mq_receive(mqh, &len, &data, false);
+
+ if (shmq_res != SHM_MQ_SUCCESS)
+ break;
+
+ if (len == 0)
+ {
+ elog(LOG, "[Apply BGW #%u] got zero-length message, stopping", pst->n);
+ break;
+ }
+ else
+ {
+ s.cursor = 0;
+ s.maxlen = -1;
+ s.data = (char *) data;
+ s.len = len;
+
+ /*
+ * We use first byte of message for additional communication between
+ * main Logical replication worker and Apply BGWorkers, so if it
+ * differs from 'w', then process it first.
+ */
+ switch (pq_getmsgbyte(&s))
+ {
+ /* Stream stop */
+ case 'E':
+ {
+ in_remote_transaction = false;
+
+ SpinLockAcquire(&pst->mutex);
+ pst->ready = true;
+ SpinLockRelease(&pst->mutex);
+ SetLatch(®istrant->procLatch);
+
+ elog(LOG, "[Apply BGW #%u] ended processing streaming chunk, waiting on shm_mq_receive", pst->n);
+
+ continue;
+ }
+ /* Reassign to the new transaction */
+ case 'R':
+ {
+ elog(LOG, "[Apply BGW #%u] switching from processing xid %u to xid %u",
+ pst->n, stream_xid, pst->stream_xid);
+ stream_xid = pst->stream_xid;
+
+ StartTransactionCommand();
+ BeginTransactionBlock();
+ CommitTransactionCommand();
+ StartTransactionCommand();
+
+ MySubscription = GetSubscription(MyLogicalRepWorker->subid, true);
+
+ continue;
+ }
+ /* Finished processing xact */
+ case 'F':
+ {
+ elog(LOG, "[Apply BGW #%u] finished processing xact %u", pst->n, stream_xid);
+
+ MemoryContextSwitchTo(ApplyContext);
+
+ CommitTransactionCommand();
+ EndTransactionBlock(false);
+ CommitTransactionCommand();
+
+ SpinLockAcquire(&pst->mutex);
+ pst->finished = true;
+ SpinLockRelease(&pst->mutex);
+
+ continue;
+ }
+ default:
+ break;
+ }
+
+ pq_getmsgint64(&s); // Read LSN info
+ pq_getmsgint64(&s); // TODO Do we need to process it here again somehow?
+ pq_getmsgint64(&s);
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ elog(DEBUG5, "[Apply BGW #%u] applying dispatch for action=%s",
+ pst->n, (char *) &s.data[s.cursor]);
+ apply_dispatch(&s);
+ }
+
+ MemoryContextSwitchTo(oldctx);
+ MemoryContextReset(ApplyMessageContext);
+ }
+
+ CommitTransactionCommand();
+ EndTransactionBlock(false);
+ CommitTransactionCommand();
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(ApplyContext);
+
+ SpinLockAcquire(&pst->mutex);
+ pst->finished = true;
+ // if (pst->workers_finished == pst->workers_total)
+ // {
+ // registrant = BackendPidGetProc(MyBgworkerEntry->bgw_notify_pid);
+ // if (registrant == NULL)
+ // {
+ // elog(DEBUG1, "registrant backend has exited prematurely");
+ // proc_exit(1);
+ // }
+ // SetLatch(®istrant->procLatch);
+ // }
+ SpinLockRelease(&pst->mutex);
+
+ elog(LOG, "[Apply BGW #%u] exiting", pst->n);
+
+ /* Signal main process that we are done. */
+ // ConditionVariableBroadcast(&cv);
+ SetLatch(®istrant->procLatch);
+
+ /*
+ * We're done. Explicitly detach the shared memory segment so that we
+ * don't get a resource leak warning at commit time. This will fire any
+ * on_dsm_detach callbacks we've registered, as well. Once that's done,
+ * we can go ahead and exit.
+ */
+ dsm_detach(seg);
+ proc_exit(0);
+}
+
+/*
+ * When we receive a SIGTERM, we set InterruptPending and ProcDiePending just
+ * like a normal backend. The next CHECK_FOR_INTERRUPTS() will do the right
+ * thing.
+ */
+static void
+handle_sigterm(SIGNAL_ARGS)
+{
+ int save_errno = errno;
+
+ SetLatch(MyLatch);
+
+ if (!proc_exit_inprogress)
+ {
+ InterruptPending = true;
+ ProcDiePending = true;
+ }
+
+ errno = save_errno;
+}
+
+/*
+ * Set up a dynamic shared memory segment.
+ *
+ * We set up a control region that contains a ParallelState,
+ * plus one region per message queue. There are as many message queues as
+ * the number of workers.
+ */
+static void
+setup_dsm(WorkerState *wstate)
+{
+ shm_toc_estimator e;
+ int toc_key = 0;
+ Size segsize;
+ dsm_segment *seg;
+ shm_toc *toc;
+ ParallelState *pst;
+ shm_mq *mq;
+ int64 queue_size = 160000000; /* 16 MB for now */
+
+ /* Ensure a valid queue size. */
+ if (queue_size < 0 || ((uint64) queue_size) < shm_mq_minimum_size)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("queue size must be at least %zu bytes",
+ shm_mq_minimum_size)));
+ if (queue_size != ((Size) queue_size))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("queue size overflows size_t")));
+
+ /*
+ * Estimate how much shared memory we need.
+ *
+ * Because the TOC machinery may choose to insert padding of oddly-sized
+ * requests, we must estimate each chunk separately.
+ *
+ * We need one key to register the location of the header, and we need
+ * nworkers keys to track the locations of the message queues.
+ */
+ shm_toc_initialize_estimator(&e);
+ shm_toc_estimate_chunk(&e, sizeof(ParallelState));
+ shm_toc_estimate_chunk(&e, (Size) queue_size);
+
+ shm_toc_estimate_keys(&e, 1 + 1);
+ segsize = shm_toc_estimate(&e);
+
+ /* Create the shared memory segment and establish a table of contents. */
+ seg = dsm_create(shm_toc_estimate(&e), 0);
+ toc = shm_toc_create(PG_LOGICAL_APPLY_SHM_MAGIC, dsm_segment_address(seg),
+ segsize);
+
+ /* Set up the header region. */
+ pst = shm_toc_allocate(toc, sizeof(ParallelState));
+ SpinLockInit(&pst->mutex);
+ pst->attached = false;
+ pst->ready = false;
+ pst->finished = false;
+ pst->database_id = MyDatabaseId;
+ pst->subid = MyLogicalRepWorker->subid;
+ pst->stream_xid = stream_xid;
+ pst->authenticated_user_id = GetAuthenticatedUserId();
+ pst->n = nworkers + 1;
+ // ConditionVariableInit(&pst->cv);
+
+ shm_toc_insert(toc, toc_key++, pst);
+
+ /* Set up one message queue per worker, plus one. */
+ mq = shm_mq_create(shm_toc_allocate(toc, (Size) queue_size),
+ (Size) queue_size);
+ shm_toc_insert(toc, toc_key++, mq);
+ shm_mq_set_sender(mq, MyProc);
+
+ /* Attach the queues. */
+ wstate->mq_handle = shm_mq_attach(mq, seg, wstate->handle);
+
+ /* Return results to caller. */
+ wstate->dsm_seg = seg;
+ wstate->pstate = pst;
+}
+
+/*
+ * Register background workers.
+ */
+static void
+setup_background_worker(WorkerState *wstate)
+{
+ MemoryContext oldcontext;
+ BackgroundWorker worker;
+
+ elog(LOG, "setting up apply worker #%u", nworkers + 1);
+
+ /*
+ * TOCHECK: We need the worker_state object and the background worker handles to
+ * which it points to be allocated in TopMemoryContext rather than
+ * ApplyMessageContext; otherwise, they'll be destroyed before the on_dsm_detach
+ * hooks run.
+ */
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+
+ setup_dsm(wstate);
+
+ /*
+ * Arrange to kill all the workers if we abort before all workers are
+ * finished hooking themselves up to the dynamic shared memory segment.
+ *
+ * If we die after all the workers have finished hooking themselves up to
+ * the dynamic shared memory segment, we'll mark the two queues to which
+ * we're directly connected as detached, and the worker(s) connected to
+ * those queues will exit, marking any other queues to which they are
+ * connected as detached. This will cause any as-yet-unaware workers
+ * connected to those queues to exit in their turn, and so on, until
+ * everybody exits.
+ *
+ * But suppose the workers which are supposed to connect to the queues to
+ * which we're directly attached exit due to some error before they
+ * actually attach the queues. The remaining workers will have no way of
+ * knowing this. From their perspective, they're still waiting for those
+ * workers to start, when in fact they've already died.
+ */
+ on_dsm_detach(wstate->dsm_seg, cleanup_background_worker,
+ PointerGetDatum(wstate));
+
+ /* Configure a worker. */
+ MemSet(&worker, 0, sizeof(BackgroundWorker));
+
+ worker.bgw_flags = BGWORKER_SHMEM_ACCESS |
+ BGWORKER_BACKEND_DATABASE_CONNECTION;
+ worker.bgw_start_time = BgWorkerStart_ConsistentState;
+ worker.bgw_restart_time = BGW_NEVER_RESTART;
+ worker.bgw_notify_pid = MyProcPid;
+ sprintf(worker.bgw_library_name, "postgres");
+ sprintf(worker.bgw_function_name, "LogicalApplyBgwMain");
+
+ worker.bgw_main_arg = UInt32GetDatum(dsm_segment_handle(wstate->dsm_seg));
+
+ /* Register the workers. */
+ snprintf(worker.bgw_name, BGW_MAXLEN,
+ "logical replication apply worker #%u for subscription %u",
+ nworkers + 1, MySubscription->oid);
+ if (!RegisterDynamicBackgroundWorker(&worker, &wstate->handle))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("could not register background process"),
+ errhint("You may need to increase max_worker_processes.")));
+
+ /* All done. */
+ MemoryContextSwitchTo(oldcontext);
+
+ /* Wait for worker to become ready. */
+ wait_for_worker(wstate);
+
+ /*
+ * Once we reach this point, all workers are ready. We no longer need to
+ * kill them if we die; they'll die on their own as the message queues
+ * shut down.
+ */
+ cancel_on_dsm_detach(wstate->dsm_seg, cleanup_background_worker,
+ PointerGetDatum(wstate));
+
+ nworkers += 1;
+}
+
+static void
+cleanup_background_worker(dsm_segment *seg, Datum arg)
+{
+ WorkerState *wstate = (WorkerState *) DatumGetPointer(arg);
+
+ TerminateBackgroundWorker(wstate->handle);
+}
+
+static void
+wait_for_worker(WorkerState *wstate)
+{
+ bool result = false;
+
+ for (;;)
+ {
+ // ConditionVariable cv;
+ bool ready;
+
+ /* If the worker is ready, we have succeeded. */
+ SpinLockAcquire(&wstate->pstate->mutex);
+ ready = wstate->pstate->ready;
+ // cv = wstate->pstate->cv;
+ SpinLockRelease(&wstate->pstate->mutex);
+ if (ready)
+ {
+ result = true;
+ break;
+ }
+
+ /* If any workers (or the postmaster) have died, we have failed. */
+ if (!check_worker_status(wstate))
+ {
+ result = false;
+ break;
+ }
+
+ /* Wait for the workers to wake us up. */
+ // ConditionVariableSleep(&cv, WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Wait to be signalled. */
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Reset the latch so we don't spin. */
+ ResetLatch(MyLatch);
+
+ /* An interrupt may have occurred while we were waiting. */
+ CHECK_FOR_INTERRUPTS();
+ }
+
+ // ConditionVariableCancelSleep();
+
+ if (!result)
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
+ errmsg("one or more background workers failed to start")));
+}
+
+static bool
+check_worker_status(WorkerState *wstate)
+{
+ BgwHandleStatus status;
+ pid_t pid;
+
+ status = GetBackgroundWorkerPid(wstate->handle, &pid);
+ if (status == BGWH_STOPPED || status == BGWH_POSTMASTER_DIED)
+ return false;
+
+ /* Otherwise, things still look OK. */
+ return true;
+}
+
+static void
+wait_for_worker_to_finish(WorkerState *wstate)
+{
+ elog(LOG, "waiting for apply worker #%u to finish processing xid %u",
+ wstate->pstate->n, wstate->xid);
+
+ for (;;)
+ {
+ // ConditionVariable cv;
+ bool finished;
+
+ /* If the worker is finished, we have succeeded. */
+ SpinLockAcquire(&wstate->pstate->mutex);
+ finished = wstate->pstate->finished;
+ // cv = wstate->pstate->cv;
+ SpinLockRelease(&wstate->pstate->mutex);
+ if (finished)
+ {
+ break;
+ }
+
+ /* Wait for the workers to wake us up. */
+ // ConditionVariableSleep(&cv, WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Wait to be signalled. */
+ WaitLatch(MyLatch, WL_LATCH_SET | WL_EXIT_ON_PM_DEATH, 0,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY);
+
+ /* Reset the latch so we don't spin. */
+ ResetLatch(MyLatch);
+
+ /* An interrupt may have occurred while we were waiting. */
+ CHECK_FOR_INTERRUPTS();
+ }
+}
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index bc45194..342e9fd 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -839,6 +839,7 @@ typedef enum
WAIT_EVENT_HASH_GROW_BUCKETS_ALLOCATING,
WAIT_EVENT_HASH_GROW_BUCKETS_ELECTING,
WAIT_EVENT_HASH_GROW_BUCKETS_REINSERTING,
+ WAIT_EVENT_LOGICAL_APPLY_WORKER_READY,
WAIT_EVENT_LOGICAL_SYNC_DATA,
WAIT_EVENT_LOGICAL_SYNC_STATE_CHANGE,
WAIT_EVENT_MQ_INTERNAL,
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index bf02cbc..479c2e2 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -122,12 +122,10 @@ extern TransactionId logicalrep_read_stream_stop(StringInfo in);
extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
XLogRecPtr commit_lsn);
-extern TransactionId logicalrep_read_stream_commit(StringInfo out,
- LogicalRepCommitData *commit_data);
+extern void logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
TransactionId subxid);
-extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
- TransactionId *subxid);
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h
index e9524ae..30ad402 100644
--- a/src/include/replication/logicalworker.h
+++ b/src/include/replication/logicalworker.h
@@ -13,6 +13,7 @@
#define LOGICALWORKER_H
extern void ApplyWorkerMain(Datum main_arg);
+extern void LogicalApplyBgwMain(Datum main_arg);
extern bool IsLogicalWorker(void);
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-10-10 10:27 ` Dilip Kumar <dilipbalaut@gmail.com>
2 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-10-10 10:27 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have attempted to test the performance of (Stream + Spill) vs
> (Stream + BGW pool) and I can see the similar gain what Alexey had
> shown[1].
>
> In addition to this, I have rebased the latest patchset [2] without
> the two-phase logical decoding patch set.
>
> Test results:
> I have repeated the same test as Alexy[1] for 1kk and 1kk data and
> here is my result
> Stream + Spill
> N time on master(sec) Total xact time (sec)
> 1kk 6 21
> 3kk 18 55
>
> Stream + BGW pool
> N time on master(sec) Total xact time (sec)
> 1kk 6 13
> 3kk 19 35
>
> Patch details:
> All the patches are the same as posted on [2] except
> 1. 0006-Gracefully-handle-concurrent-aborts-of-uncommitted -> I have
> removed the handling of error which is specific for 2PC
Here[1], I mentioned that I have removed the 2PC changes from
this[0006] patch but mistakenly I attached the original patch itself
instead of the modified version. So attaching the modified version of
only this patch other patches are the same.
> 2. 0007-Implement-streaming-mode-in-ReorderBuffer -> Rebased without 2PC
> 3. 0009-Extend-the-concurrent-abort-handling-for-in-progress -> New
> patch to handle concurrent abort error for the in-progress transaction
> and also add handling for the sub transaction's abort.
> 4. v3-0014-BGWorkers-pool-for-streamed-transactions-apply -> Rebased
> Alexey's patch
[1] https://www.postgresql.org/message-id/CAFiTN-vHoksqvV4BZ0479NhugGe4QHq_ezngNdDd-YRQ_2cwug%40mail.gma...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0006-Gracefully-handle-concurrent-aborts-of-uncommitted-t.patch (application/octet-stream, 12.4K) download
From 9739d48a979868b912f6e1f90d4702808e061a35 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 3 Oct 2019 09:00:49 +0530
Subject: [PATCH 06/14] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++--
src/backend/utils/time/snapmgr.c | 25 +++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 119 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65..da6a6f3 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index e954482..6ce7878 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1433,16 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1714,16 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index ca4b904..3143479 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -679,7 +679,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1529,7 +1529,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1780,7 +1780,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1800,7 +1800,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Every time the CommandId is incremented, we could
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-10-22 05:16 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-10-22 05:16 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; a.kondratov@postgrespro.ru; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have attempted to test the performance of (Stream + Spill) vs
> (Stream + BGW pool) and I can see the similar gain what Alexey had
> shown[1].
>
> In addition to this, I have rebased the latest patchset [2] without
> the two-phase logical decoding patch set.
>
> Test results:
> I have repeated the same test as Alexy[1] for 1kk and 1kk data and
> here is my result
> Stream + Spill
> N time on master(sec) Total xact time (sec)
> 1kk 6 21
> 3kk 18 55
>
> Stream + BGW pool
> N time on master(sec) Total xact time (sec)
> 1kk 6 13
> 3kk 19 35
>
I think the test results for the master are missing. Also, how about
running these tests over a network (means master and subscriber are
not on the same machine)? In general, yours and Alexy's test results
show that there is merit by having workers applying such transactions.
OTOH, as noted above [1], we are also worried about the performance
of Rollbacks if we follow that approach. I am not sure how much we
need to worry about Rollabcks if commits are faster, but can we think
of recording the changes in memory and only write to a file if the
changes are above a certain threshold? I think that might help saving
I/O in many cases. I am not very sure if we do that how much
additional workers can help, but they might still help. I think we
need to do some tests and experiments to figure out what is the best
approach? What do you think?
Tomas, Alexey, do you have any thoughts on this matter? I think it is
important that we figure out the way to proceed in this patch.
[1] - https://www.postgresql.org/message-id/b25ce80e-f536-78c8-d5c8-a5df3e230785%40postgrespro.ru
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-10-22 05:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-10-22 05:31 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: a.kondratov@postgrespro.ru, Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 22, 2019 at 10:46 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have attempted to test the performance of (Stream + Spill) vs
> > (Stream + BGW pool) and I can see the similar gain what Alexey had
> > shown[1].
> >
> > In addition to this, I have rebased the latest patchset [2] without
> > the two-phase logical decoding patch set.
> >
> > Test results:
> > I have repeated the same test as Alexy[1] for 1kk and 1kk data and
> > here is my result
> > Stream + Spill
> > N time on master(sec) Total xact time (sec)
> > 1kk 6 21
> > 3kk 18 55
> >
> > Stream + BGW pool
> > N time on master(sec) Total xact time (sec)
> > 1kk 6 13
> > 3kk 19 35
> >
>
> I think the test results for the master are missing.
Yeah, That time, I was planning to compare spill vs bgworker.
Also, how about
> running these tests over a network (means master and subscriber are
> not on the same machine)?
Yeah, we should do that that will show the merit of streaming the
in-progress transactions.
In general, yours and Alexy's test results
> show that there is merit by having workers applying such transactions.
> OTOH, as noted above [1], we are also worried about the performance
> of Rollbacks if we follow that approach. I am not sure how much we
> need to worry about Rollabcks if commits are faster, but can we think
> of recording the changes in memory and only write to a file if the
> changes are above a certain threshold? I think that might help saving
> I/O in many cases. I am not very sure if we do that how much
> additional workers can help, but they might still help. I think we
> need to do some tests and experiments to figure out what is the best
> approach? What do you think?
I agree with the point. I think we might need to do some small
changes and test to see what could be the best method to handle the
streamed changes at the subscriber end.
>
> Tomas, Alexey, do you have any thoughts on this matter? I think it is
> important that we figure out the way to proceed in this patch.
>
> [1] - https://www.postgresql.org/message-id/b25ce80e-f536-78c8-d5c8-a5df3e230785%40postgrespro.ru
>
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-10-22 17:22 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-22 19:02 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-30 04:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Tomas Vondra @ 2019-10-22 17:22 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 22, 2019 at 11:01:48AM +0530, Dilip Kumar wrote:
>On Tue, Oct 22, 2019 at 10:46 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> >
>> > I have attempted to test the performance of (Stream + Spill) vs
>> > (Stream + BGW pool) and I can see the similar gain what Alexey had
>> > shown[1].
>> >
>> > In addition to this, I have rebased the latest patchset [2] without
>> > the two-phase logical decoding patch set.
>> >
>> > Test results:
>> > I have repeated the same test as Alexy[1] for 1kk and 1kk data and
>> > here is my result
>> > Stream + Spill
>> > N time on master(sec) Total xact time (sec)
>> > 1kk 6 21
>> > 3kk 18 55
>> >
>> > Stream + BGW pool
>> > N time on master(sec) Total xact time (sec)
>> > 1kk 6 13
>> > 3kk 19 35
>> >
>>
>> I think the test results for the master are missing.
>Yeah, That time, I was planning to compare spill vs bgworker.
> Also, how about
>> running these tests over a network (means master and subscriber are
>> not on the same machine)?
>
>Yeah, we should do that that will show the merit of streaming the
>in-progress transactions.
>
Which I agree it's an interesting feature, I think we need to stop
adding more stuff to this patch series - it's already complex enough, so
making it even more (unnecessary) stuff is a distraction and will make
it harder to get anything committed. Typical "scope creep".
I think the current behavior (spill to file) is sufficient for v0 and
can be improved later - that's fine. I don't think we need to bother
with comparisons to master very much, because while it might be a bit
slower in some cases, you can always disable streaming (so if there's a
regression for your workload, you can undo that).
> In general, yours and Alexy's test results
>> show that there is merit by having workers applying such transactions.
>> OTOH, as noted above [1], we are also worried about the performance
>> of Rollbacks if we follow that approach. I am not sure how much we
>> need to worry about Rollabcks if commits are faster, but can we think
>> of recording the changes in memory and only write to a file if the
>> changes are above a certain threshold? I think that might help saving
>> I/O in many cases. I am not very sure if we do that how much
>> additional workers can help, but they might still help. I think we
>> need to do some tests and experiments to figure out what is the best
>> approach? What do you think?
>I agree with the point. I think we might need to do some small
>changes and test to see what could be the best method to handle the
>streamed changes at the subscriber end.
>
>>
>> Tomas, Alexey, do you have any thoughts on this matter? I think it is
>> important that we figure out the way to proceed in this patch.
>>
>> [1] - https://www.postgresql.org/message-id/b25ce80e-f536-78c8-d5c8-a5df3e230785%40postgrespro.ru
>>
>
I think the patch should do the simplest thing possible, i.e. what it
does today. Otherwise we'll never get it committed.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2019-10-22 19:02 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-23 10:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Alexey Kondratov @ 2019-10-22 19:02 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 22.10.2019 20:22, Tomas Vondra wrote:
> On Tue, Oct 22, 2019 at 11:01:48AM +0530, Dilip Kumar wrote:
>> On Tue, Oct 22, 2019 at 10:46 AM Amit Kapila
>> <amit.kapila16@gmail.com> wrote:
>> In general, yours and Alexy's test results
>>> show that there is merit by having workers applying such transactions.
>>> OTOH, as noted above [1], we are also worried about the performance
>>> of Rollbacks if we follow that approach. I am not sure how much we
>>> need to worry about Rollabcks if commits are faster, but can we think
>>> of recording the changes in memory and only write to a file if the
>>> changes are above a certain threshold? I think that might help saving
>>> I/O in many cases. I am not very sure if we do that how much
>>> additional workers can help, but they might still help. I think we
>>> need to do some tests and experiments to figure out what is the best
>>> approach? What do you think?
>> I agree with the point. I think we might need to do some small
>> changes and test to see what could be the best method to handle the
>> streamed changes at the subscriber end.
>>
>>>
>>> Tomas, Alexey, do you have any thoughts on this matter? I think it is
>>> important that we figure out the way to proceed in this patch.
>>>
>>> [1] -
>>> https://www.postgresql.org/message-id/b25ce80e-f536-78c8-d5c8-a5df3e230785%40postgrespro.ru
>>>
>>
>
> I think the patch should do the simplest thing possible, i.e. what it
> does today. Otherwise we'll never get it committed.
>
I have to agree with Tomas, that keeping things as simple as possible
should be a main priority right now. Otherwise, the entire patch set
will pass next release cycle without being committed at least partially.
In the same time, it resolves important problem from my perspective. It
moves I/O overhead from primary to replica using large transactions
streaming, which is a nice to have feature I guess.
Later it would be possible to replace logical apply worker with
bgworkers pool in a separated patch, if we decide that it is a viable
solution. Anyway, regarding the Amit's questions:
- I doubt that maintaining a separate buffer on the apply side before
spilling to disk would help enough. We already have ReorderBuffer with
logical_work_mem limit, and if we exceeded that limit on the sender
side, then most probably we exceed it on the applier side as well,
excepting the case when this new buffer size will be significantly
higher then logical_work_mem to keep multiple open xacts.
- I still think that we should optimize database for commits, not
rollbacks. BGworkers pool is dramatically slower for rollbacks-only
load, though being at least twice as faster for commits-only. I do not
know how it will perform with real life load, but this drawback may be
inappropriate for such a general purpose database like Postgres.
- Tomas' implementation of streaming with spilling does not have this
bias between commits/aborts. However, it has a noticeable performance
drop (~x5 slower compared with master [1]) for large transaction
consisting of many small rows. Although it is not of an order of
magnitude slower.
Another thing is it that about a year ago I have found some problems
with MVCC/visibility and fixed them somehow [1]. If I get it correctly
Tomas adapted some of those fixes into his patch set, but I think that
this part should be reviewed carefully again. I would be glad to check
it, but now I am a little bit confused with all the patch set variants
in the thread. Which is the last one? Is it still dependent on 2pc decoding?
[1]
https://www.postgresql.org/message-id/flat/40c38758-04b5-74f4-c963-cf300f9e5dff%40postgrespro.ru#98d...
Thanks for moving this patch forward!
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-22 19:02 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alexey Kondratov <a.kondratov@postgrespro.ru>
@ 2019-10-23 10:50 ` Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-10-23 10:50 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Dilip Kumar <dilipbalaut@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Oct 23, 2019 at 12:32 AM Alexey Kondratov
<a.kondratov@postgrespro.ru> wrote:
>
> On 22.10.2019 20:22, Tomas Vondra wrote:
> >
> > I think the patch should do the simplest thing possible, i.e. what it
> > does today. Otherwise we'll never get it committed.
> >
>
> I have to agree with Tomas, that keeping things as simple as possible
> should be a main priority right now. Otherwise, the entire patch set
> will pass next release cycle without being committed at least partially.
> In the same time, it resolves important problem from my perspective. It
> moves I/O overhead from primary to replica using large transactions
> streaming, which is a nice to have feature I guess.
>
> Later it would be possible to replace logical apply worker with
> bgworkers pool in a separated patch, if we decide that it is a viable
> solution. Anyway, regarding the Amit's questions:
>
> - I doubt that maintaining a separate buffer on the apply side before
> spilling to disk would help enough. We already have ReorderBuffer with
> logical_work_mem limit, and if we exceeded that limit on the sender
> side, then most probably we exceed it on the applier side as well,
>
I think on the sender side, the limit is for un-filtered changes
(which means on the ReorderBuffer which has all the changes) whereas,
on the receiver side, we will only have the requested changes which
can make a difference?
> excepting the case when this new buffer size will be significantly
> higher then logical_work_mem to keep multiple open xacts.
>
I am not sure but I think we can have different controlling parameters
on the subscriber-side.
> - I still think that we should optimize database for commits, not
> rollbacks. BGworkers pool is dramatically slower for rollbacks-only
> load, though being at least twice as faster for commits-only. I do not
> know how it will perform with real life load, but this drawback may be
> inappropriate for such a general purpose database like Postgres.
>
> - Tomas' implementation of streaming with spilling does not have this
> bias between commits/aborts. However, it has a noticeable performance
> drop (~x5 slower compared with master [1]) for large transaction
> consisting of many small rows. Although it is not of an order of
> magnitude slower.
>
Did you ever identify the reason why it was slower in that case? I
can see the numbers shared by you and Dilip which shows that the
BGWorker pool is a really good idea and will work great for
commit-mostly workload whereas the numbers without that are not very
encouraging, maybe we have not benchmarked enough. This is the reason
I am trying to see if we can do something to get the benefits similar
to what is shown by your idea.
I am not against doing something simple for the first version and then
enhance it later, but it won't be good if we commit it with regression
in some typical cases and depend on the user to use it when it seems
favorable to its case. Also, sometimes it becomes difficult to
generate enthusiasm to enhance the feature once the main patch is
committed. I am not telling that always happens or will happen in
this case. It is better if we put some energy and get things as good
as possible in the first go itself. I am as much interested as you,
Tomas or others are, otherwise, I wouldn't have spent a lot of time on
this to disentangle it from 2PC patch which seems to get stalled due
to lack of interest.
> Another thing is it that about a year ago I have found some problems
> with MVCC/visibility and fixed them somehow [1]. If I get it correctly
> Tomas adapted some of those fixes into his patch set, but I think that
> this part should be reviewed carefully again.
>
Agreed, I have read your emails and could see that you have done very
good work on this project along with Tomas. But unfortunately, it
didn't get committed. At this stage, we are working on just the first
part of the patch which is to allow the data to spill once it crosses
the logical_decoding_work_mem on the master side. I think we need
more problems to discuss and solve once that is done.
> I would be glad to check
> it, but now I am a little bit confused with all the patch set variants
> in the thread. Which is the last one? Is it still dependent on 2pc decoding?
>
I think the latest patches posted by Dilip are not dependent on
logical decoding, but I haven't studied them yet. You can find those
at [1][2]. As per discussion in this thread, we are also trying to
see if we can make some part of the patch-series committed first, the
latest patches corresponding to which are posted at [3].
[1] - https://www.postgresql.org/message-id/CAFiTN-vHoksqvV4BZ0479NhugGe4QHq_ezngNdDd-YRQ_2cwug%40mail.gma...
[2] - https://www.postgresql.org/message-id/CAFiTN-vT%2B42xRbkw%3DhBnp44XkAyZaKZVA5hcvAMsYth3rk7vhg%40mail...
[3] - https://www.postgresql.org/message-id/CAFiTN-vkFB0RBEjVkLWhdgTYShSrSu3kCYObMghgXEwKA1FXRA%40mail.gma...
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2019-10-30 04:08 ` vignesh C <vignesh21@gmail.com>
2019-10-30 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 11:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: vignesh C @ 2019-10-30 04:08 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Oct 22, 2019 at 10:52 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> I think the patch should do the simplest thing possible, i.e. what it
> does today. Otherwise we'll never get it committed.
>
I found a couple of crashes while reviewing and testing flushing of
open transaction data:
Issue 1:
#0 0x00007f22c5722337 in raise () from /lib64/libc.so.6
#1 0x00007f22c5723a28 in abort () from /lib64/libc.so.6
#2 0x0000000000ec5390 in ExceptionalCondition
(conditionName=0x10ea814 "!dlist_is_empty(head)", errorType=0x10ea804
"FailedAssertion",
fileName=0x10ea7e0 "../../../../src/include/lib/ilist.h",
lineNumber=458) at assert.c:54
#3 0x0000000000b4fb91 in dlist_tail_element_off (head=0x19e4db8,
off=64) at ../../../../src/include/lib/ilist.h:458
#4 0x0000000000b546d0 in ReorderBufferAbortOld (rb=0x191b6b0,
oldestRunningXid=3834) at reorderbuffer.c:1966
#5 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x19af990,
buf=0x7ffcbc26dc50) at decode.c:332
#6 0x0000000000b3c208 in LogicalDecodingProcessRecord (ctx=0x19af990,
record=0x19afc50) at decode.c:121
#7 0x0000000000b7109e in XLogSendLogical () at walsender.c:2845
#8 0x0000000000b6f5e4 in WalSndLoop (send_data=0xb70f77
<XLogSendLogical>) at walsender.c:2199
#9 0x0000000000b6c7e1 in StartLogicalReplication (cmd=0x1983168) at
walsender.c:1128
#10 0x0000000000b6da6f in exec_replication_command
(cmd_string=0x18f70a0 "START_REPLICATION SLOT \"sub1\" LOGICAL 0/0
(proto_version '1', publication_names '\"pub1\"')")
at walsender.c:1545
Issue 2:
#0 0x00007f1d7ddc4337 in raise () from /lib64/libc.so.6
#1 0x00007f1d7ddc5a28 in abort () from /lib64/libc.so.6
#2 0x0000000000ec4e1d in ExceptionalCondition
(conditionName=0x10ead30 "txn->final_lsn != InvalidXLogRecPtr",
errorType=0x10ea284 "FailedAssertion",
fileName=0x10ea2d0 "reorderbuffer.c", lineNumber=3052) at assert.c:54
#3 0x0000000000b577e0 in ReorderBufferRestoreCleanup (rb=0x2ae36b0,
txn=0x2bafb08) at reorderbuffer.c:3052
#4 0x0000000000b52b1c in ReorderBufferCleanupTXN (rb=0y x2ae36b0,
txn=0x2bafb08) at reorderbuffer.c:1318
#5 0x0000000000b5279d in ReorderBufferCleanupTXN (rb=0x2ae36b0,
txn=0x2b9d778) at reorderbuffer.c:1257
#6 0x0000000000b5475c in ReorderBufferAbortOld (rb=0x2ae36b0,
oldestRunningXid=3835) at reorderbuffer.c:1973
#7 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x2b676d0,
buf=0x7ffcbc74cc00) at decode.c:332
#8 0x0000000000b3c208 in LogicalDecodingProcessRecord (ctx=0x2b676d0,
record=0x2b67990) at decode.c:121
#9 0x0000000000b70b2b in XLogSendLogical () at walsender.c:2845
These failures come randomly.
I'm not able to reproduce this issue with simple test case.
I have attached the test case which I used to test.
I will further try to find a scenario which could reproduce consistently.
Posting it so that it can help someone in identifying the problem
parallelly through code review by experts.
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
Attachments:
mix_data_test.c (text/x-c-code, 7.1K) download
#include <stdio.h>
#include <stdlib.h>
#include <libpq-fe.h>
#define LARGEDATATHREADS 32
#define SMALLDATATHREADS 50
void do_exit(PGconn *conn, PGresult *res)
{
fprintf(stderr, "%s\n", PQerrorMessage(conn));
PQclear(res);
PQfinish(conn);
exit(1);
}
void initsubscriber()
{
PGconn *conn = PQconnectdb("user=user1 dbname=testdb port=5433");
if (PQstatus(conn) == CONNECTION_BAD)
{
fprintf(stderr, "Connection to database failed: %s\n",
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
PGresult *res = PQexec(conn, "DROP TABLE IF EXISTS perftest_smalldata");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "CREATE TABLE perftest_smalldata(c1 varchar(10), c2 varchar(10), c3 varchar(10), c4 varchar(10), c5 varchar(10))");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "DROP TABLE IF EXISTS perftest");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "CREATE TABLE perftest(c1 varchar(100), c2 varchar(100), c3 varchar(100), c4 varchar(100), c5 varchar(100))");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "create subscription sub1 connection 'host=127.0.0.1 port=5432 dbname=testdb user=user1 password=123' publication pub1");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
PQfinish(conn);
}
void initpublisher()
{
PGconn *conn = PQconnectdb("user=user1 dbname=testdb");
if (PQstatus(conn) == CONNECTION_BAD)
{
fprintf(stderr, "Connection to database failed: %s\n",
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
PGresult *res = PQexec(conn, "DROP TABLE IF EXISTS perftest_smalldata");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "CREATE TABLE perftest_smalldata(c1 varchar(10), c2 varchar(10), c3 varchar(10), c4 varchar(10), c5 varchar(10))");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "DROP TABLE IF EXISTS perftest");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "CREATE TABLE perftest(c1 varchar(100), c2 varchar(100), c3 varchar(100), c4 varchar(100), c5 varchar(100))");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
res = PQexec(conn, "CREATE PUBLICATION pub1 FOR TABLE perftest, perftest_smalldata WITH (publish='insert,update,delete')");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
PQfinish(conn);
}
void* smalldatathreadfunc(void *recordcount)
{
unsigned int reccount = *((unsigned int *) recordcount);
int count;
PGresult *res;
PGconn *conn = PQconnectdb("user=user1 dbname=testdb");
if (PQstatus(conn) == CONNECTION_BAD)
{
fprintf(stderr, "Connection to database failed: %s\n",
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
res = PQexec(conn, "BEGIN");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
/* Load data */
int subtxncount = reccount/64;
int savepointcount = 0;
for (count = 0; count < reccount; count++)
{
res = PQexec(conn, "insert into perftest_smalldata values('asafadfafa', 'asafadfafa', 'asafadfafa', 'asafadfafa', 'asafadfafa')");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
do_exit(conn, res);
PQclear(res);
if (count % subtxncount == 0)
{
char savepointstr[128] = {0};
savepointcount++;
sprintf(savepointstr, "savepoint sp%d", savepointcount);
res = PQexec(conn, savepointstr);
if (PQresultStatus(res) != PGRES_COMMAND_OK)
do_exit(conn, res);
PQclear(res);
}
}
printf("%d subtransaction created\n", savepointcount);
printf("Keeping txn open for 60 seconds\n");
sleep(60);
res = PQexec(conn, "SELECT * FROM perftest_smalldata");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
printf("No data retrieved\n");
PQclear(res);
PQfinish(conn);
exit(1);
}
int rows = PQntuples(res);
printf("Record count = %d\n", rows);
PQclear(res);
PQfinish(conn);
return 0;
}
void* largedatathreadfunc(void *recordcount)
{
unsigned int reccount = *((unsigned int *) recordcount);
int count;
PGresult *res;
PGconn *conn = PQconnectdb("user=user1 dbname=testdb");
if (PQstatus(conn) == CONNECTION_BAD)
{
fprintf(stderr, "Connection to database failed: %s\n",
PQerrorMessage(conn));
PQfinish(conn);
exit(1);
}
res = PQexec(conn, "BEGIN");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
{
do_exit(conn, res);
}
PQclear(res);
/* Load data */
for (count = 0; count < reccount; count++)
{
res = PQexec(conn, "insert into perftest values(\
'asafadfadfdasdasdafsafasfafafsasfasafsdafsfasfasdsdfasfdsfdsasdfsasaddfsadsdasdfssafdsfsfssasdfss',\
'asafadfadfdasdasdafsafasfafafsasfasafsdafsfasfasdsdfasfdsfdsasdfsasaddfsadsdasdfssafdsfsfssasdfss',\
'asafadfadfdasdasdafsafasfafafsasfasafsdafsfasfasdsdfasfdsfdsasdfsasaddfsadsdasdfssafdsfsfssasdfss',\
'asafadfadfdasdasdafsafasfafafsasfasafsdafsfasfasdsdfasfdsfdsasdfsasaddfsadsdasdfssafdsfsfssasdfss',\
'asafadfadfdasdasdafsafasfafafsasfasafsdafsfasfasdsdfasfdsfdsasdfsasaddfsadsdasdfssafdsfsfssasdfss')");
if (PQresultStatus(res) != PGRES_COMMAND_OK)
do_exit(conn, res);
PQclear(res);
}
printf("Keeping txn open for 60 seconds\n");
sleep(60);
res = PQexec(conn, "SELECT * FROM perftest");
if (PQresultStatus(res) != PGRES_TUPLES_OK)
{
printf("No data retrieved\n");
PQclear(res);
PQfinish(conn);
exit(1);
}
int rows = PQntuples(res);
printf("Record count = %d\n", rows);
PQclear(res);
PQfinish(conn);
return 0;
}
int main()
{
pthread_t largedata_thread_id[LARGEDATATHREADS];
pthread_t smalldata_thread_id[SMALLDATATHREADS];
int i;
int recordcount;
initpublisher();
initsubscriber();
for(i=0; i < LARGEDATATHREADS; i++)
{
recordcount = 2500;
pthread_create( &largedata_thread_id[i], NULL, largedatathreadfunc, &recordcount);
}
for(i=0; i < SMALLDATATHREADS; i++)
{
recordcount = 6000;
pthread_create( &smalldata_thread_id[i], NULL, smalldatathreadfunc, &recordcount);
}
for(i=0; i < LARGEDATATHREADS; i++)
{
pthread_join( largedata_thread_id[i], NULL);
}
for(i=0; i < SMALLDATATHREADS; i++)
{
pthread_join( smalldata_thread_id[i], NULL);
}
}
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-30 04:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
@ 2019-10-30 07:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 09:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-10-30 07:48 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Oct 30, 2019 at 9:38 AM vignesh C <vignesh21@gmail.com> wrote:
>
I have noticed one more problem in the logic of setting the logical
decoding work mem from the create subscription command. Suppose in
subscription command we don't give the work mem then it sends the
garbage value to the walsender and the walsender overwrite its value
with the garbage value. After investigating a bit I have found the
reason for the same.
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
I think the problem is we are unconditionally sending the work_mem as
part of the CREATE REPLICATION SLOT, without checking whether it's
valid or not.
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -71,6 +71,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
Another problem is that there is no handling if the subform->subworkmem is NULL.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-30 04:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
2019-10-30 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-11-04 09:13 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-04 10:02 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Kuntal Ghosh @ 2019-11-04 09:13 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
Hello hackers,
I've done some performance testing of this feature. Following is my
test case (taken from an earlier thread):
postgres=# CREATE TABLE large_test (num1 bigint, num2 double
precision, num3 double precision);
postgres=# \timing on
postgres=# EXPLAIN (ANALYZE, BUFFERS) INSERT INTO large_test (num1,
num2, num3) SELECT round(random()*10), random(), random()*142 FROM
generate_series(1, 1000000) s(i);
I've kept the publisher and subscriber in two different system.
HEAD:
With 1000000 tuples,
Execution Time: 2576.821 ms, Time: 9.632.158 ms (00:09.632), Spill count: 245
With 10000000 tuples (10 times more),
Execution Time: 30359.509 ms, Time: 95261.024 ms (01:35.261), Spill count: 2442
With the memory accounting patch, following are the performance results:
With 100000 tuples,
logical_decoding_work_mem=64kB, Execution Time: 2414.371 ms, Time:
9648.223 ms (00:09.648), Spill count: 2315
logical_decoding_work_mem=64MB, Execution Time: 2477.830 ms, Time:
9895.161 ms (00:09.895), Spill count 3
With 1000000 tuples (10 times more),
logical_decoding_work_mem=64kB, Execution Time: 38259.227 ms, Time:
105761.978 ms (01:45.762), Spill count: 23149
logical_decoding_work_mem=64MB, Execution Time: 24624.639 ms, Time:
89985.342 ms (01:29.985), Spill count: 23
With logical decoding of in-progress transactions patch and with
streaming on, following are the performance results:
With 100000 tuples,
logical_decoding_work_mem=64kB, Execution Time: 2674.034 ms, Time:
20779.601 ms (00:20.780)
logical_decoding_work_mem=64MB, Execution Time: 2062.404 ms, Time:
9559.953 ms (00:09.560)
With 1000000 tuples (10 times more),
logical_decoding_work_mem=64kB, Execution Time: 26949.588 ms, Time:
196261.892 ms (03:16.262)
logical_decoding_work_mem=64MB, Execution Time: 27084.403 ms, Time:
90079.286 ms (01:30.079)
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-30 04:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
2019-10-30 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 09:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
@ 2019-11-04 10:02 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 10:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-04 10:02 UTC (permalink / raw)
To: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 4, 2019 at 2:43 PM Kuntal Ghosh <kuntalghosh.2007@gmail.com> wrote:
>
> Hello hackers,
>
> I've done some performance testing of this feature. Following is my
> test case (taken from an earlier thread):
>
> postgres=# CREATE TABLE large_test (num1 bigint, num2 double
> precision, num3 double precision);
> postgres=# \timing on
> postgres=# EXPLAIN (ANALYZE, BUFFERS) INSERT INTO large_test (num1,
> num2, num3) SELECT round(random()*10), random(), random()*142 FROM
> generate_series(1, 1000000) s(i);
>
> I've kept the publisher and subscriber in two different system.
>
> HEAD:
> With 1000000 tuples,
> Execution Time: 2576.821 ms, Time: 9.632.158 ms (00:09.632), Spill count: 245
> With 10000000 tuples (10 times more),
> Execution Time: 30359.509 ms, Time: 95261.024 ms (01:35.261), Spill count: 2442
>
> With the memory accounting patch, following are the performance results:
> With 100000 tuples,
> logical_decoding_work_mem=64kB, Execution Time: 2414.371 ms, Time:
> 9648.223 ms (00:09.648), Spill count: 2315
> logical_decoding_work_mem=64MB, Execution Time: 2477.830 ms, Time:
> 9895.161 ms (00:09.895), Spill count 3
> With 1000000 tuples (10 times more),
> logical_decoding_work_mem=64kB, Execution Time: 38259.227 ms, Time:
> 105761.978 ms (01:45.762), Spill count: 23149
> logical_decoding_work_mem=64MB, Execution Time: 24624.639 ms, Time:
> 89985.342 ms (01:29.985), Spill count: 23
>
> With logical decoding of in-progress transactions patch and with
> streaming on, following are the performance results:
> With 100000 tuples,
> logical_decoding_work_mem=64kB, Execution Time: 2674.034 ms, Time:
> 20779.601 ms (00:20.780)
> logical_decoding_work_mem=64MB, Execution Time: 2062.404 ms, Time:
> 9559.953 ms (00:09.560)
> With 1000000 tuples (10 times more),
> logical_decoding_work_mem=64kB, Execution Time: 26949.588 ms, Time:
> 196261.892 ms (03:16.262)
> logical_decoding_work_mem=64MB, Execution Time: 27084.403 ms, Time:
> 90079.286 ms (01:30.079)
So your result shows that with "streaming on", performance is
degrading? By any chance did you try to see where is the bottleneck?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-30 04:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
2019-10-30 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 09:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-04 10:02 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-11-04 10:05 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-12 10:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alexey Kondratov <a.kondratov@postgrespro.ru>
0 siblings, 1 reply; 536+ messages in thread
From: Kuntal Ghosh @ 2019-11-04 10:05 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; a.kondratov@postgrespro.ru, Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 4, 2019 at 3:32 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> So your result shows that with "streaming on", performance is
> degrading? By any chance did you try to see where is the bottleneck?
>
Right. But, as we increase the logical_decoding_work_mem, the
performance improves. I've not analyzed the bottleneck yet. I'm
looking into the same.
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-30 04:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
2019-10-30 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 09:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-04 10:02 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 10:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
@ 2019-11-12 10:42 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-11-12 11:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Alexey Kondratov @ 2019-11-12 10:42 UTC (permalink / raw)
To: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Dilip Kumar <dilipbalaut@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 04.11.2019 13:05, Kuntal Ghosh wrote:
> On Mon, Nov 4, 2019 at 3:32 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> So your result shows that with "streaming on", performance is
>> degrading? By any chance did you try to see where is the bottleneck?
>>
> Right. But, as we increase the logical_decoding_work_mem, the
> performance improves. I've not analyzed the bottleneck yet. I'm
> looking into the same.
My guess is that 64 kB is just too small value. In the table schema used
for tests every rows takes at least 24 bytes for storing column values.
Thus, with this logical_decoding_work_mem value the limit should be hit
after about 2500+ rows, or about 400 times during transaction of 1000000
rows size.
It is just too frequent, while ReorderBufferStreamTXN includes a whole
bunch of logic, e.g. it always starts internal transaction:
/*
* Decoding needs access to syscaches et al., which in turn use
* heavyweight locks and such. Thus we need to have enough state around to
* keep track of those. The easiest way is to simply use a transaction
* internally. That also allows us to easily enforce that nothing writes
* to the database by checking for xid assignments. ...
*/
Also it issues separated stream_start/stop messages around each streamed
transaction chunk. So if streaming starts and stops too frequently it
adds additional overhead and may even interfere with current in-progress
transaction.
If I get it correctly, then it is rather expected with too small values
of logical_decoding_work_mem. Probably it may be optimized, but I am not
sure that it is worth doing right now.
Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-30 04:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
2019-10-30 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 09:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-04 10:02 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 10:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-12 10:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alexey Kondratov <a.kondratov@postgrespro.ru>
@ 2019-11-12 11:34 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Kuntal Ghosh @ 2019-11-12 11:34 UTC (permalink / raw)
To: Alexey Kondratov <a.kondratov@postgrespro.ru>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Nov 12, 2019 at 4:12 PM Alexey Kondratov
<a.kondratov@postgrespro.ru> wrote:
>
> On 04.11.2019 13:05, Kuntal Ghosh wrote:
> > On Mon, Nov 4, 2019 at 3:32 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >> So your result shows that with "streaming on", performance is
> >> degrading? By any chance did you try to see where is the bottleneck?
> >>
> > Right. But, as we increase the logical_decoding_work_mem, the
> > performance improves. I've not analyzed the bottleneck yet. I'm
> > looking into the same.
>
> My guess is that 64 kB is just too small value. In the table schema used
> for tests every rows takes at least 24 bytes for storing column values.
> Thus, with this logical_decoding_work_mem value the limit should be hit
> after about 2500+ rows, or about 400 times during transaction of 1000000
> rows size.
>
> It is just too frequent, while ReorderBufferStreamTXN includes a whole
> bunch of logic, e.g. it always starts internal transaction:
>
> /*
> * Decoding needs access to syscaches et al., which in turn use
> * heavyweight locks and such. Thus we need to have enough state around to
> * keep track of those. The easiest way is to simply use a transaction
> * internally. That also allows us to easily enforce that nothing writes
> * to the database by checking for xid assignments. ...
> */
>
> Also it issues separated stream_start/stop messages around each streamed
> transaction chunk. So if streaming starts and stops too frequently it
> adds additional overhead and may even interfere with current in-progress
> transaction.
>
Yeah, I've also found the same. With stream_start/stop message, it
writes 1 byte of checksum and 4 bytes of number of sub-transactions
which increases the write amplification significantly.
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-30 04:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
@ 2019-11-04 11:51 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-05 05:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-06 11:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2019-11-04 11:51 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Dilip Kumar <dilipbalaut@gmail.com>; Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Oct 30, 2019 at 9:38 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Tue, Oct 22, 2019 at 10:52 PM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> >
> > I think the patch should do the simplest thing possible, i.e. what it
> > does today. Otherwise we'll never get it committed.
> >
> I found a couple of crashes while reviewing and testing flushing of
> open transaction data:
>
Thanks for doing these tests. However, I don't think these issues are
anyway related to this patch. It seems to be base code issues
manifested by this patch. See my analysis below.
> Issue 1:
> #0 0x00007f22c5722337 in raise () from /lib64/libc.so.6
> #1 0x00007f22c5723a28 in abort () from /lib64/libc.so.6
> #2 0x0000000000ec5390 in ExceptionalCondition
> (conditionName=0x10ea814 "!dlist_is_empty(head)", errorType=0x10ea804
> "FailedAssertion",
> fileName=0x10ea7e0 "../../../../src/include/lib/ilist.h",
> lineNumber=458) at assert.c:54
> #3 0x0000000000b4fb91 in dlist_tail_element_off (head=0x19e4db8,
> off=64) at ../../../../src/include/lib/ilist.h:458
> #4 0x0000000000b546d0 in ReorderBufferAbortOld (rb=0x191b6b0,
> oldestRunningXid=3834) at reorderbuffer.c:1966
> #5 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x19af990,
> buf=0x7ffcbc26dc50) at decode.c:332
>
This seems to be the problem of base code where we abort immediately
after serializing the changes because in that case, the changes list
will be empty. I think you can try to reproduce it via the debugger
or by hacking the code such that it serializes after every change and
then if you abort after one change, it should hit this problem.
>
> Issue 2:
> #0 0x00007f1d7ddc4337 in raise () from /lib64/libc.so.6
> #1 0x00007f1d7ddc5a28 in abort () from /lib64/libc.so.6
> #2 0x0000000000ec4e1d in ExceptionalCondition
> (conditionName=0x10ead30 "txn->final_lsn != InvalidXLogRecPtr",
> errorType=0x10ea284 "FailedAssertion",
> fileName=0x10ea2d0 "reorderbuffer.c", lineNumber=3052) at assert.c:54
> #3 0x0000000000b577e0 in ReorderBufferRestoreCleanup (rb=0x2ae36b0,
> txn=0x2bafb08) at reorderbuffer.c:3052
> #4 0x0000000000b52b1c in ReorderBufferCleanupTXN (rb=0y x2ae36b0,
> txn=0x2bafb08) at reorderbuffer.c:1318
> #5 0x0000000000b5279d in ReorderBufferCleanupTXN (rb=0x2ae36b0,
> txn=0x2b9d778) at reorderbuffer.c:1257
> #6 0x0000000000b5475c in ReorderBufferAbortOld (rb=0x2ae36b0,
> oldestRunningXid=3835) at reorderbuffer.c:1973
>
This seems to be again the problem with base code as we don't update
the final_lsn for subtransactions during ReorderBufferAbortOld. This
can also be reproduced with some hacking in code or via debugger in a
similar way as explained for the previous problem but with a
difference that there must be subtransaction involved in this case.
> #7 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x2b676d0,
> buf=0x7ffcbc74cc00) at decode.c:332
> #8 0x0000000000b3c208 in LogicalDecodingProcessRecord (ctx=0x2b676d0,
> record=0x2b67990) at decode.c:121
> #9 0x0000000000b70b2b in XLogSendLogical () at walsender.c:2845
>
> These failures come randomly.
> I'm not able to reproduce this issue with simple test case.
Yeah, it appears to be difficult to reproduce unless you hack the code
to serialize every change or use debugger to forcefully flush the
changes every time.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-30 04:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
2019-11-04 11:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-11-05 05:38 ` Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-11-05 05:38 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 4, 2019 at 5:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Oct 30, 2019 at 9:38 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, Oct 22, 2019 at 10:52 PM Tomas Vondra
> > <tomas.vondra@2ndquadrant.com> wrote:
> > >
> > > I think the patch should do the simplest thing possible, i.e. what it
> > > does today. Otherwise we'll never get it committed.
> > >
> > I found a couple of crashes while reviewing and testing flushing of
> > open transaction data:
> >
>
> Thanks for doing these tests. However, I don't think these issues are
> anyway related to this patch. It seems to be base code issues
> manifested by this patch. See my analysis below.
>
> > Issue 1:
> > #0 0x00007f22c5722337 in raise () from /lib64/libc.so.6
> > #1 0x00007f22c5723a28 in abort () from /lib64/libc.so.6
> > #2 0x0000000000ec5390 in ExceptionalCondition
> > (conditionName=0x10ea814 "!dlist_is_empty(head)", errorType=0x10ea804
> > "FailedAssertion",
> > fileName=0x10ea7e0 "../../../../src/include/lib/ilist.h",
> > lineNumber=458) at assert.c:54
> > #3 0x0000000000b4fb91 in dlist_tail_element_off (head=0x19e4db8,
> > off=64) at ../../../../src/include/lib/ilist.h:458
> > #4 0x0000000000b546d0 in ReorderBufferAbortOld (rb=0x191b6b0,
> > oldestRunningXid=3834) at reorderbuffer.c:1966
> > #5 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x19af990,
> > buf=0x7ffcbc26dc50) at decode.c:332
> >
>
> This seems to be the problem of base code where we abort immediately
> after serializing the changes because in that case, the changes list
> will be empty. I think you can try to reproduce it via the debugger
> or by hacking the code such that it serializes after every change and
> then if you abort after one change, it should hit this problem.
>
I think you might need to kill the server after all changes are
serialized otherwise normal abort will hit the ReorderBufferAbort and
that will remove your ReorderBufferTXN entry and you will never hit
this case.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-30 04:08 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
2019-11-04 11:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-11-06 11:03 ` vignesh C <vignesh21@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: vignesh C @ 2019-11-06 11:03 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Dilip Kumar <dilipbalaut@gmail.com>; Alexey Kondratov <a.kondratov@postgrespro.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 4, 2019 at 5:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Oct 30, 2019 at 9:38 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Tue, Oct 22, 2019 at 10:52 PM Tomas Vondra
> > <tomas.vondra@2ndquadrant.com> wrote:
> > >
> > > I think the patch should do the simplest thing possible, i.e. what it
> > > does today. Otherwise we'll never get it committed.
> > >
> > I found a couple of crashes while reviewing and testing flushing of
> > open transaction data:
> >
>
> Thanks for doing these tests. However, I don't think these issues are
> anyway related to this patch. It seems to be base code issues
> manifested by this patch. See my analysis below.
>
> > Issue 1:
> > #0 0x00007f22c5722337 in raise () from /lib64/libc.so.6
> > #1 0x00007f22c5723a28 in abort () from /lib64/libc.so.6
> > #2 0x0000000000ec5390 in ExceptionalCondition
> > (conditionName=0x10ea814 "!dlist_is_empty(head)", errorType=0x10ea804
> > "FailedAssertion",
> > fileName=0x10ea7e0 "../../../../src/include/lib/ilist.h",
> > lineNumber=458) at assert.c:54
> > #3 0x0000000000b4fb91 in dlist_tail_element_off (head=0x19e4db8,
> > off=64) at ../../../../src/include/lib/ilist.h:458
> > #4 0x0000000000b546d0 in ReorderBufferAbortOld (rb=0x191b6b0,
> > oldestRunningXid=3834) at reorderbuffer.c:1966
> > #5 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x19af990,
> > buf=0x7ffcbc26dc50) at decode.c:332
> >
>
> This seems to be the problem of base code where we abort immediately
> after serializing the changes because in that case, the changes list
> will be empty. I think you can try to reproduce it via the debugger
> or by hacking the code such that it serializes after every change and
> then if you abort after one change, it should hit this problem.
>
> >
> > Issue 2:
> > #0 0x00007f1d7ddc4337 in raise () from /lib64/libc.so.6
> > #1 0x00007f1d7ddc5a28 in abort () from /lib64/libc.so.6
> > #2 0x0000000000ec4e1d in ExceptionalCondition
> > (conditionName=0x10ead30 "txn->final_lsn != InvalidXLogRecPtr",
> > errorType=0x10ea284 "FailedAssertion",
> > fileName=0x10ea2d0 "reorderbuffer.c", lineNumber=3052) at assert.c:54
> > #3 0x0000000000b577e0 in ReorderBufferRestoreCleanup (rb=0x2ae36b0,
> > txn=0x2bafb08) at reorderbuffer.c:3052
> > #4 0x0000000000b52b1c in ReorderBufferCleanupTXN (rb=0y x2ae36b0,
> > txn=0x2bafb08) at reorderbuffer.c:1318
> > #5 0x0000000000b5279d in ReorderBufferCleanupTXN (rb=0x2ae36b0,
> > txn=0x2b9d778) at reorderbuffer.c:1257
> > #6 0x0000000000b5475c in ReorderBufferAbortOld (rb=0x2ae36b0,
> > oldestRunningXid=3835) at reorderbuffer.c:1973
> >
>
> This seems to be again the problem with base code as we don't update
> the final_lsn for subtransactions during ReorderBufferAbortOld. This
> can also be reproduced with some hacking in code or via debugger in a
> similar way as explained for the previous problem but with a
> difference that there must be subtransaction involved in this case.
>
> > #7 0x0000000000b3ca03 in DecodeStandbyOp (ctx=0x2b676d0,
> > buf=0x7ffcbc74cc00) at decode.c:332
> > #8 0x0000000000b3c208 in LogicalDecodingProcessRecord (ctx=0x2b676d0,
> > record=0x2b67990) at decode.c:121
> > #9 0x0000000000b70b2b in XLogSendLogical () at walsender.c:2845
> >
> > These failures come randomly.
> > I'm not able to reproduce this issue with simple test case.
>
> Yeah, it appears to be difficult to reproduce unless you hack the code
> to serialize every change or use debugger to forcefully flush the
> changes every time.
>
Thanks Amit for your analysis, I was able to reproduce the above issue
consistently by making some code changes and with help of debugger. I
did one change so that it flushes every time instead of flushing after
the buffer size exceeds the logical_decoding_work_mem, attached one of
the transactions and called abort. When the server restarts after
abort, this problem occurs consistently. I could reproduce the issue
with base code also. It seems like this issue is not an issue of
0001-Add-logical_decoding_work_mem-to-limit-ReorderBuffer patch and
exists from base code. I will post the issue in hackers with details.
Regards,
Vignesh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-11-13 12:25 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-13 12:25 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
As mentioned by me a few days back that the first patch in this series
is ready to go [1] (I am hoping Tomas will pick it up), so I have
started the review of other patches
Review/Questions on 0002-Immediately-WAL-log-assignments.patch
-------------------------------------------------------------------------------------------------
1. This patch adds the top_xid in WAL whenever the first time WAL for
a subtransaction XID is written to correctly decode the changes of
in-progress transaction. This patch also removes logging and applying
WAL for XLOG_XACT_ASSIGNMENT which might have some effect. As replay
of that, it prunes KnownAssignedXids to prevent overflow of that
array. See comments in procarray.c (KnownAssignedTransactionIds
sub-module). Can you please explain how after removing the WAL for
XLOG_XACT_ASSIGNMENT, we will handle that or I am missing something
and there is no impact of same?
2.
+#define XLOG_INCLUDE_INVALS 0x08 /* include invalidations */
This doesn't seem to be used in this patch.
[1] - https://www.postgresql.org/message-id/CAA4eK1JM0%3DRwODZQrn8DTQ3dbcb9xwKDdHCmVOryAk_xoKf9Nw%40mail.g...
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-11-14 04:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-14 04:07 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Nov 13, 2019 at 5:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
>
> As mentioned by me a few days back that the first patch in this series
> is ready to go [1] (I am hoping Tomas will pick it up), so I have
> started the review of other patches
>
> Review/Questions on 0002-Immediately-WAL-log-assignments.patch
> -------------------------------------------------------------------------------------------------
> 1. This patch adds the top_xid in WAL whenever the first time WAL for
> a subtransaction XID is written to correctly decode the changes of
> in-progress transaction. This patch also removes logging and applying
> WAL for XLOG_XACT_ASSIGNMENT which might have some effect. As replay
> of that, it prunes KnownAssignedXids to prevent overflow of that
> array. See comments in procarray.c (KnownAssignedTransactionIds
> sub-module). Can you please explain how after removing the WAL for
> XLOG_XACT_ASSIGNMENT, we will handle that or I am missing something
> and there is no impact of same?
It seems like a problem to me as well. One option could be that
since now we are adding the top transaction id in the first WAL of the
subtransaction we can directly update the pg_subtrans and avoid adding
sub transaction id in the KnownAssignedXids and mark it as
lastOverflowedXid. But, I don't think we should go in that direction
otherwise it will impact the performance of visibility check on the
hot-standby. Let's see what Tomas has in mind.
>
> 2.
> +#define XLOG_INCLUDE_INVALS 0x08 /* include invalidations */
>
> This doesn't seem to be used in this patch.
>
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-11-14 06:40 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-14 06:40 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 14, 2019 at 9:37 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Nov 13, 2019 at 5:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> >
> > As mentioned by me a few days back that the first patch in this series
> > is ready to go [1] (I am hoping Tomas will pick it up), so I have
> > started the review of other patches
> >
> > Review/Questions on 0002-Immediately-WAL-log-assignments.patch
> > -------------------------------------------------------------------------------------------------
> > 1. This patch adds the top_xid in WAL whenever the first time WAL for
> > a subtransaction XID is written to correctly decode the changes of
> > in-progress transaction. This patch also removes logging and applying
> > WAL for XLOG_XACT_ASSIGNMENT which might have some effect. As replay
> > of that, it prunes KnownAssignedXids to prevent overflow of that
> > array. See comments in procarray.c (KnownAssignedTransactionIds
> > sub-module). Can you please explain how after removing the WAL for
> > XLOG_XACT_ASSIGNMENT, we will handle that or I am missing something
> > and there is no impact of same?
>
> It seems like a problem to me as well. One option could be that
> since now we are adding the top transaction id in the first WAL of the
> subtransaction we can directly update the pg_subtrans and avoid adding
> sub transaction id in the KnownAssignedXids and mark it as
> lastOverflowedXid.
>
Hmm, I am not sure if we can do that easily because I think in
RecordKnownAssignedTransactionIds, we add those based on the gap via
KnownAssignedXidsAdd and only remove them later while applying WAL for
XLOG_XACT_ASSIGNMENT. I think if we really want to go in this
direction then for each WAL record we need to check if it has
XLR_BLOCK_ID_TOPLEVEL_XID set and then call function
ProcArrayApplyXidAssignment() with the required information. I think
this line of attack has WAL overhead both on master whenever
subtransactions are involved and also on hot-standby for doing the
work for each subtransaction separately. The WAL apply needs to
acquire and release PROCArrayLock in exclusive mode for each
subtransaction whereas now it does it once for
PGPROC_MAX_CACHED_SUBXIDS number of subtransactions which can conflict
with queries running on standby.
The other idea could be that we keep the current XLOG_XACT_ASSIGNMENT
mechanism (WAL logging and apply of same on hot-standby) as it is and
additionally log top_xid the first time when WAL is written for a
subtransaction only when wal_level >= WAL_LEVEL_LOGICAL. Then use the
same for logical decoding. The advantage of this approach is that we
will incur the overhead of additional transactionid only when required
especially not with default server configuration.
Thoughts?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-11-14 10:09 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-14 10:09 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 14, 2019 at 12:10 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Nov 14, 2019 at 9:37 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Wed, Nov 13, 2019 at 5:55 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > >
> > > As mentioned by me a few days back that the first patch in this series
> > > is ready to go [1] (I am hoping Tomas will pick it up), so I have
> > > started the review of other patches
> > >
> > > Review/Questions on 0002-Immediately-WAL-log-assignments.patch
> > > -------------------------------------------------------------------------------------------------
> > > 1. This patch adds the top_xid in WAL whenever the first time WAL for
> > > a subtransaction XID is written to correctly decode the changes of
> > > in-progress transaction. This patch also removes logging and applying
> > > WAL for XLOG_XACT_ASSIGNMENT which might have some effect. As replay
> > > of that, it prunes KnownAssignedXids to prevent overflow of that
> > > array. See comments in procarray.c (KnownAssignedTransactionIds
> > > sub-module). Can you please explain how after removing the WAL for
> > > XLOG_XACT_ASSIGNMENT, we will handle that or I am missing something
> > > and there is no impact of same?
> >
> > It seems like a problem to me as well. One option could be that
> > since now we are adding the top transaction id in the first WAL of the
> > subtransaction we can directly update the pg_subtrans and avoid adding
> > sub transaction id in the KnownAssignedXids and mark it as
> > lastOverflowedXid.
> >
>
> Hmm, I am not sure if we can do that easily because I think in
> RecordKnownAssignedTransactionIds, we add those based on the gap via
> KnownAssignedXidsAdd and only remove them later while applying WAL for
> XLOG_XACT_ASSIGNMENT. I think if we really want to go in this
> direction then for each WAL record we need to check if it has
> XLR_BLOCK_ID_TOPLEVEL_XID set and then call function
> ProcArrayApplyXidAssignment() with the required information. I think
> this line of attack has WAL overhead both on master whenever
> subtransactions are involved and also on hot-standby for doing the
> work for each subtransaction separately. The WAL apply needs to
> acquire and release PROCArrayLock in exclusive mode for each
> subtransaction whereas now it does it once for
> PGPROC_MAX_CACHED_SUBXIDS number of subtransactions which can conflict
> with queries running on standby.
Right
>
> The other idea could be that we keep the current XLOG_XACT_ASSIGNMENT
> mechanism (WAL logging and apply of same on hot-standby) as it is and
> additionally log top_xid the first time when WAL is written for a
> subtransaction only when wal_level >= WAL_LEVEL_LOGICAL. Then use the
> same for logical decoding. The advantage of this approach is that we
> will incur the overhead of additional transactionid only when required
> especially not with default server configuration.
>
> Thoughts?
The idea seems reasonable to me.
Apart from this, I have another question in
0003-Issue-individual-invalidations-with-wal_level-logical.patch
@@ -543,6 +588,18 @@ RegisterSnapshotInvalidation(Oid dbId, Oid relId)
{
AddSnapshotInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.sn.id = SHAREDINVALSNAPSHOT_ID;
+ msg.sn.dbId = dbId;
+ msg.sn.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
I am not sure why do we need to explicitly WAL log the snapshot
invalidation? because this is logged for invalidating the catalog
snapshot and for logical decoding we use HistoricSnapshot, not the
catalog snapshot. I might be missing something?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-11-15 10:20 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-15 10:20 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 14, 2019 at 3:40 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
>
> Apart from this, I have another question in
> 0003-Issue-individual-invalidations-with-wal_level-logical.patch
>
> @@ -543,6 +588,18 @@ RegisterSnapshotInvalidation(Oid dbId, Oid relId)
> {
> AddSnapshotInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
> dbId, relId);
> +
> + /* Issue an invalidation WAL record (when wal_level=logical) */
> + if (XLogLogicalInfoActive())
> + {
> + SharedInvalidationMessage msg;
> +
> + msg.sn.id = SHAREDINVALSNAPSHOT_ID;
> + msg.sn.dbId = dbId;
> + msg.sn.relId = relId;
> +
> + LogLogicalInvalidations(1, &msg, false);
> + }
> }
>
> I am not sure why do we need to explicitly WAL log the snapshot
> invalidation? because this is logged for invalidating the catalog
> snapshot and for logical decoding we use HistoricSnapshot, not the
> catalog snapshot.
>
I think it has been logged because without this patch as well we log
all the invalidation messages at commit time and process them during
decoding. However, I agree that this particular invalidation message
is not required for logical decoding for the reason you mentioned. I
think as we are explicitly logging invalidations, so it is better to
avoid this if we can.
Few other comments on this patch:
1.
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
Here, why are we executing messages individually? Can't we just
follow what we do in DecodeCommit which is to record the invalidations
in ReorderBufferTXN as we encounter them and then allow them to
execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
reason why we don't do ReorderBufferXidSetCatalogChanges when we
receive any invalidation message?
2.
@@ -3025,8 +3073,8 @@ ReorderBufferRestoreChange(ReorderBuffer *rb,
ReorderBufferTXN *txn,
* although we don't check the memory limit when restoring the changes in
* this branch (we only do that when initially queueing the changes after
* decoding), because we will release the changes later, and that will
- * update the accounting too (subtracting the size from the counters).
- * And we don't want to underflow there.
+ * update the accounting too (subtracting the size from the counters). And
+ * we don't want to underflow there.
*/
This seems like an unrelated change.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-11-15 10:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-15 10:31 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Nov 14, 2019 at 3:40 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> >
> > Apart from this, I have another question in
> > 0003-Issue-individual-invalidations-with-wal_level-logical.patch
> >
> > @@ -543,6 +588,18 @@ RegisterSnapshotInvalidation(Oid dbId, Oid relId)
> > {
> > AddSnapshotInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
> > dbId, relId);
> > +
> > + /* Issue an invalidation WAL record (when wal_level=logical) */
> > + if (XLogLogicalInfoActive())
> > + {
> > + SharedInvalidationMessage msg;
> > +
> > + msg.sn.id = SHAREDINVALSNAPSHOT_ID;
> > + msg.sn.dbId = dbId;
> > + msg.sn.relId = relId;
> > +
> > + LogLogicalInvalidations(1, &msg, false);
> > + }
> > }
> >
> > I am not sure why do we need to explicitly WAL log the snapshot
> > invalidation? because this is logged for invalidating the catalog
> > snapshot and for logical decoding we use HistoricSnapshot, not the
> > catalog snapshot.
> >
>
> I think it has been logged because without this patch as well we log
> all the invalidation messages at commit time and process them during
> decoding. However, I agree that this particular invalidation message
> is not required for logical decoding for the reason you mentioned. I
> think as we are explicitly logging invalidations, so it is better to
> avoid this if we can.
Ok
>
> Few other comments on this patch:
> 1.
> + case REORDER_BUFFER_CHANGE_INVALIDATION:
> +
> + /*
> + * Execute the invalidation message locally.
> + *
> + * XXX Do we need to care about relcacheInitFileInval and
> + * the other fields added to ReorderBufferChange, or just
> + * about the message itself?
> + */
> + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> + break;
>
> Here, why are we executing messages individually? Can't we just
> follow what we do in DecodeCommit which is to record the invalidations
> in ReorderBufferTXN as we encounter them and then allow them to
> execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> reason why we don't do ReorderBufferXidSetCatalogChanges when we
> receive any invalidation message?
IMHO, the reason is that in DecodeCommit, we get all the invalidation
at one time so, at REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID, we don't
know which invalidation message to execute so for being safe we have
to execute all. But, since we are logging all invalidation
individually, we exactly know at this stage which cache to invalidate.
So it is better to only invalidate required cache not all.
>
> 2.
> @@ -3025,8 +3073,8 @@ ReorderBufferRestoreChange(ReorderBuffer *rb,
> ReorderBufferTXN *txn,
> * although we don't check the memory limit when restoring the changes in
> * this branch (we only do that when initially queueing the changes after
> * decoding), because we will release the changes later, and that will
> - * update the accounting too (subtracting the size from the counters).
> - * And we don't want to underflow there.
> + * update the accounting too (subtracting the size from the counters). And
> + * we don't want to underflow there.
> */
>
> This seems like an unrelated change.
Indeed.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-11-15 10:49 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-15 10:49 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Few other comments on this patch:
> > 1.
> > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > +
> > + /*
> > + * Execute the invalidation message locally.
> > + *
> > + * XXX Do we need to care about relcacheInitFileInval and
> > + * the other fields added to ReorderBufferChange, or just
> > + * about the message itself?
> > + */
> > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > + break;
> >
> > Here, why are we executing messages individually? Can't we just
> > follow what we do in DecodeCommit which is to record the invalidations
> > in ReorderBufferTXN as we encounter them and then allow them to
> > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > receive any invalidation message?
> IMHO, the reason is that in DecodeCommit, we get all the invalidation
> at one time so, at REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID, we don't
> know which invalidation message to execute so for being safe we have
> to execute all. But, since we are logging all invalidation
> individually, we exactly know at this stage which cache to invalidate.
> So it is better to only invalidate required cache not all.
>
In that case, invalidations can be processed multiple times, the first
time when these individual WAL logs for invalidation are processed and
then later at commit time when we accumulate all invalidation messages
and then execute them for REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID.
Can we avoid to execute invalidations from other places after this
patch which also includes executing them as part of XLOG_INVALIDATIONS
processing?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-11-18 11:32 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-18 11:32 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Few other comments on this patch:
> > > 1.
> > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > +
> > > + /*
> > > + * Execute the invalidation message locally.
> > > + *
> > > + * XXX Do we need to care about relcacheInitFileInval and
> > > + * the other fields added to ReorderBufferChange, or just
> > > + * about the message itself?
> > > + */
> > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > + break;
> > >
> > > Here, why are we executing messages individually? Can't we just
> > > follow what we do in DecodeCommit which is to record the invalidations
> > > in ReorderBufferTXN as we encounter them and then allow them to
> > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > receive any invalidation message?
I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
commit. Because this is required to add any committed transaction to
the snapshot if it has done any catalog changes. So I think there is
no point in setting that flag every time we get an invalidation
message.
> > IMHO, the reason is that in DecodeCommit, we get all the invalidation
> > at one time so, at REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID, we don't
> > know which invalidation message to execute so for being safe we have
> > to execute all. But, since we are logging all invalidation
> > individually, we exactly know at this stage which cache to invalidate.
> > So it is better to only invalidate required cache not all.
> >
>
> In that case, invalidations can be processed multiple times, the first
> time when these individual WAL logs for invalidation are processed and
> then later at commit time when we accumulate all invalidation messages
> and then execute them for REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID.
> Can we avoid to execute invalidations from other places after this
> patch which also includes executing them as part of XLOG_INVALIDATIONS
> processing?
I think we can avoid invalidation which is done as part of
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. I need to further
investigate the invalidation which is done as part of
XLOG_INVALIDATIONS.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-11-19 11:53 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-11-19 11:53 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Nov 18, 2019 at 5:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > Few other comments on this patch:
> > > > 1.
> > > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > > +
> > > > + /*
> > > > + * Execute the invalidation message locally.
> > > > + *
> > > > + * XXX Do we need to care about relcacheInitFileInval and
> > > > + * the other fields added to ReorderBufferChange, or just
> > > > + * about the message itself?
> > > > + */
> > > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > > + break;
> > > >
> > > > Here, why are we executing messages individually? Can't we just
> > > > follow what we do in DecodeCommit which is to record the invalidations
> > > > in ReorderBufferTXN as we encounter them and then allow them to
> > > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > > receive any invalidation message?
>
> I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
> commit. Because this is required to add any committed transaction to
> the snapshot if it has done any catalog changes.
>
Hmm, this is also used to build cid hash map (see
ReorderBufferBuildTupleCidHash) which we need to use while streaming
changes for the in-progress transactions. So, I think that it would
be required earlier (before commit) as well.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-11-20 05:45 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-20 05:45 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Nov 19, 2019 at 5:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Nov 18, 2019 at 5:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Few other comments on this patch:
> > > > > 1.
> > > > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > > > +
> > > > > + /*
> > > > > + * Execute the invalidation message locally.
> > > > > + *
> > > > > + * XXX Do we need to care about relcacheInitFileInval and
> > > > > + * the other fields added to ReorderBufferChange, or just
> > > > > + * about the message itself?
> > > > > + */
> > > > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > > > + break;
> > > > >
> > > > > Here, why are we executing messages individually? Can't we just
> > > > > follow what we do in DecodeCommit which is to record the invalidations
> > > > > in ReorderBufferTXN as we encounter them and then allow them to
> > > > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > > > receive any invalidation message?
> >
> > I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
> > commit. Because this is required to add any committed transaction to
> > the snapshot if it has done any catalog changes.
> >
>
> Hmm, this is also used to build cid hash map (see
> ReorderBufferBuildTupleCidHash) which we need to use while streaming
> changes for the in-progress transactions. So, I think that it would
> be required earlier (before commit) as well.
>
Oh right, I guess I missed that part.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-11-20 14:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-20 14:52 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Nov 20, 2019 at 11:15 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Nov 19, 2019 at 5:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Nov 18, 2019 at 5:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > >
> > > > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > >
> > > > > >
> > > > > > Few other comments on this patch:
> > > > > > 1.
> > > > > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > > > > +
> > > > > > + /*
> > > > > > + * Execute the invalidation message locally.
> > > > > > + *
> > > > > > + * XXX Do we need to care about relcacheInitFileInval and
> > > > > > + * the other fields added to ReorderBufferChange, or just
> > > > > > + * about the message itself?
> > > > > > + */
> > > > > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > > > > + break;
> > > > > >
> > > > > > Here, why are we executing messages individually? Can't we just
> > > > > > follow what we do in DecodeCommit which is to record the invalidations
> > > > > > in ReorderBufferTXN as we encounter them and then allow them to
> > > > > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > > > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > > > > receive any invalidation message?
> > >
> > > I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
> > > commit. Because this is required to add any committed transaction to
> > > the snapshot if it has done any catalog changes.
> > >
> >
> > Hmm, this is also used to build cid hash map (see
> > ReorderBufferBuildTupleCidHash) which we need to use while streaming
> > changes for the in-progress transactions. So, I think that it would
> > be required earlier (before commit) as well.
> >
> Oh right, I guess I missed that part.
Attached a new rebased version of the patch set. I have fixed all
the issues discussed up-thread and agreed upon.
Pending Issues:
1. The default value of the logical_decoding_work_mem is set to 64kb
in test_decoding/logical.conf. So we need to change the expected
output files for the test decoding module.
2. Need to complete the patch for concurrent abort handling of the
(sub)transaction. There are some pending issues with the existing
patch[1].
[1] https://www.postgresql.org/message-id/CAFiTN-ud98kWHCo2YKS55H8rGw3_A7ESyssHwU0xPU6KJsoy6A%40mail.gma...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
0002-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.2K) download
From 33439d94691e636a4439d4e234d76fc9916b8359 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH 02/13] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So instead we write the assignment info into WAL immediately, as
part of the next WAL record (to minimize overhead).
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 8fe38c3..4a853f3 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index aa9dca0..a8a8084 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 7f24f0c..4ef2661 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1071,6 +1071,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1109,6 +1110,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index bc532d0..897b755 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 42b76cb..0c7daf4 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252..b492d3e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 1bbee38..c37a83d 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -148,6 +148,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -243,6 +245,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54..bcfba0a 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
0001-Track-statistics-for-spilling-of-changes-from-Reorde.patch (application/octet-stream, 11.5K) download
From 45bdc8b9116844c2a458110363b14e820911bccb Mon Sep 17 00:00:00 2001
From: Amit Kapila <akapila@postgresql.org>
Date: Sat, 16 Nov 2019 18:24:00 +0530
Subject: [PATCH 01/13] Track statistics for spilling of changes from
ReorderBuffer.
This adds the statistics about transactions spilled to disk from
ReorderBuffer. Users can query the pg_stat_replication view to check
these stats.
Author: Tomas Vondra, with bug-fixes and minor changes by Dilip Kumar
Reviewed-by: Amit Kapila
Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
---
doc/src/sgml/monitoring.sgml | 20 ++++++++++++
src/backend/catalog/system_views.sql | 5 ++-
src/backend/replication/logical/reorderbuffer.c | 12 +++++++
src/backend/replication/walsender.c | 42 +++++++++++++++++++++++--
src/include/catalog/pg_proc.dat | 6 ++--
src/include/replication/reorderbuffer.h | 11 +++++++
src/include/replication/walsender_private.h | 5 +++
src/test/regress/expected/rules.out | 7 +++--
8 files changed, 100 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 901fee9..a3c5f86 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1972,6 +1972,26 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<entry><type>timestamp with time zone</type></entry>
<entry>Send time of last reply message received from standby server</entry>
</row>
+ <row>
+ <entry><structfield>spill_bytes</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Amount of decoded transaction data spilled to disk.</entry>
+ </row>
+ <row>
+ <entry><structfield>spill_txns</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Number of transactions spilled to disk after the memory used by
+ logical decoding exceeds <literal>logical_decoding_work_mem</literal>. The
+ counter gets incremented both for toplevel transactions and
+ subtransactions.</entry>
+ </row>
+ <row>
+ <entry><structfield>spill_count</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry>Number of times transactions were spilled to disk. Transactions
+ may get spilled repeatedly, and this counter gets incremented on every
+ such invocation.</entry>
+ </row>
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 4456fef..f7800f0 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -776,7 +776,10 @@ CREATE VIEW pg_stat_replication AS
W.replay_lag,
W.sync_priority,
W.sync_state,
- W.reply_time
+ W.reply_time,
+ W.spill_txns,
+ W.spill_count,
+ W.spill_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index d82a5f1..53affeb 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -308,6 +308,10 @@ ReorderBufferAllocate(void)
buffer->outbufsize = 0;
buffer->size = 0;
+ buffer->spillCount = 0;
+ buffer->spillTxns = 0;
+ buffer->spillBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -2415,6 +2419,7 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
int fd = -1;
XLogSegNo curOpenSegNo = 0;
Size spilled = 0;
+ Size size = txn->size;
elog(DEBUG2, "spill %u changes in XID %u to disk",
(uint32) txn->nentries_mem, txn->xid);
@@ -2473,6 +2478,13 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
spilled++;
}
+ /* update the statistics */
+ rb->spillCount += 1;
+ rb->spillBytes += size;
+
+ /* Don't consider already serialized transaction. */
+ rb->spillTxns += txn->serialized ? 0 : 1;
+
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 7f56715..fa75872 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -248,6 +248,7 @@ static void LagTrackerWrite(XLogRecPtr lsn, TimestampTz local_flush_time);
static TimeOffset LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now);
static bool TransactionIdInRecentPast(TransactionId xid, uint32 epoch);
+static void UpdateSpillStats(LogicalDecodingContext *ctx);
static void XLogRead(WALSegmentContext *segcxt, char *buf, XLogRecPtr startptr, Size count);
@@ -1261,7 +1262,8 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
/*
* LogicalDecodingContext 'update_progress' callback.
*
- * Write the current position to the lag tracker (see XLogSendPhysical).
+ * Write the current position to the lag tracker (see XLogSendPhysical),
+ * and update the spill statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1280,6 +1282,11 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
LagTrackerWrite(lsn, now);
sendTime = now;
+
+ /*
+ * Update statistics about transactions that spilled to disk.
+ */
+ UpdateSpillStats(ctx);
}
/*
@@ -2318,6 +2325,9 @@ InitWalSenderSlot(void)
walsnd->state = WALSNDSTATE_STARTUP;
walsnd->latch = &MyProc->procLatch;
walsnd->replyTime = 0;
+ walsnd->spillTxns = 0;
+ walsnd->spillCount = 0;
+ walsnd->spillBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3219,7 +3229,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 12
+#define PG_STAT_GET_WAL_SENDERS_COLS 15
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3274,6 +3284,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int pid;
WalSndState state;
TimestampTz replyTime;
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3294,6 +3307,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
applyLag = walsnd->applyLag;
priority = walsnd->sync_standby_priority;
replyTime = walsnd->replyTime;
+ spillTxns = walsnd->spillTxns;
+ spillCount = walsnd->spillCount;
+ spillBytes = walsnd->spillBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3375,6 +3391,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
nulls[11] = true;
else
values[11] = TimestampTzGetDatum(replyTime);
+
+ /* spill to disk */
+ values[12] = Int64GetDatum(spillTxns);
+ values[13] = Int64GetDatum(spillCount);
+ values[14] = Int64GetDatum(spillBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3611,3 +3632,20 @@ LagTrackerRead(int head, XLogRecPtr lsn, TimestampTz now)
Assert(time != 0);
return now - time;
}
+
+static void
+UpdateSpillStats(LogicalDecodingContext *ctx)
+{
+ ReorderBuffer *rb = ctx->reorder;
+
+ SpinLockAcquire(&MyWalSnd->mutex);
+
+ MyWalSnd->spillTxns = rb->spillTxns;
+ MyWalSnd->spillCount = rb->spillCount;
+ MyWalSnd->spillBytes = rb->spillBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
+
+ SpinLockRelease(&MyWalSnd->mutex);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58ea5b9..fa0a2a1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7c94d92..0867ee9 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -402,6 +402,17 @@ struct ReorderBuffer
/* memory accounting */
Size size;
+
+ /*
+ * Statistics about transactions spilled to disk.
+ *
+ * A single transaction may be spilled repeatedly, which is why we keep
+ * two different counters. For spilling, the transaction counter includes
+ * both toplevel transactions and subtransactions.
+ */
+ int64 spillCount; /* spill-to-disk invocation counter */
+ int64 spillTxns; /* number of transactions spilled to disk */
+ int64 spillBytes; /* amount of data spilled to disk */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 0dd6d1c..a6b3205 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -80,6 +80,11 @@ typedef struct WalSnd
* Timestamp of the last message received from standby.
*/
TimestampTz replyTime;
+
+ /* Statistics for transactions spilled to disk. */
+ int64 spillTxns;
+ int64 spillCount;
+ int64 spillBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index abe3a43..c9cc569 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1952,9 +1952,12 @@ pg_stat_replication| SELECT s.pid,
w.replay_lag,
w.sync_priority,
w.sync_state,
- w.reply_time
+ w.reply_time,
+ w.spill_txns,
+ w.spill_count,
+ w.spill_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
0003-Issue-individual-invalidations-with-wal_level-logica.patch (application/octet-stream, 15.8K) download
From e4d3e144b8babaf08be669c17b577c5348e0e3cc Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH 03/13] Issue individual invalidations with wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 52 +++++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 +++++++++
src/backend/replication/logical/reorderbuffer.c | 56 +++++++++++++++++---
src/backend/utils/cache/inval.c | 69 +++++++++++++++++++++++++
src/include/access/xact.h | 18 ++++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 4c411c5..6cfd6af 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,46 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 4a853f3..a42d11f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 897b755..9bcefb6 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 53affeb..b1feff3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -464,6 +464,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1804,17 +1805,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2209,6 +2216,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2656,6 +2695,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2752,6 +2792,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3027,6 +3068,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9..0682c55 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1546,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 0c7daf4..5bd3893 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0867ee9..6a7187b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structure.patch (application/octet-stream, 8.1K) download
From d750bf1746a2cd61ff9d72ff690baf400512605c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH 05/13] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++---------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b1feff3..3422939 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -732,7 +732,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -752,7 +752,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -775,7 +775,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -835,7 +835,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -851,7 +851,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1061,7 +1061,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1090,7 +1090,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1256,7 +1256,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1304,7 +1304,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1319,7 +1319,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1336,7 +1336,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1970,7 +1970,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2118,7 +2118,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2297,7 +2297,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2314,7 +2314,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2334,7 +2334,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2522,12 +2522,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b4be2b..19c7bac 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
0004-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From f2d26607b79cacc54c89b70a8f88d6194e814f26 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH 04/13] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4b..9c77791 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..fc4ad65 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting. At
+ that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 7e06615..b88b585 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 31c796b..d95d1b9 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -81,6 +81,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f..a305462 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6a7187b..5b4be2b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
0006-Gracefully-handle-concurrent-aborts-of-uncommitted-t.patch (application/octet-stream, 12.5K) download
From fc83f3bbe01cd7e807033e5a9b2ef8dd48b8f5ca Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH 06/13] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 9 +++--
src/backend/utils/time/snapmgr.c | 25 +++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 120 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65..da6a6f3 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0128bb3..2a60a73 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1433,16 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1714,16 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3422939..bda4a1c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -683,7 +683,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1533,7 +1533,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1784,7 +1784,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1804,7 +1804,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
@@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
0007-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 44.3K) download
From 0251050a1ba6ece6ee3a8826b5e1ac5a136c8e39 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:42:31 +0530
Subject: [PATCH 07/13] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 1075 ++++++++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1112 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 537e681..76a105a 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index bda4a1c..0ab3191 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -362,6 +408,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -759,6 +808,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -855,6 +931,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -978,7 +1057,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1006,6 +1085,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1020,6 +1102,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1235,6 +1320,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
+/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1327,33 +1616,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1403,6 +1763,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1476,6 +1846,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1515,6 +1898,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_lsn = origin_lsn;
/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
* ReorderBufferCommitChild will have transferred any snapshots from
@@ -1549,6 +1948,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1930,6 +2340,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2014,6 +2431,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2149,8 +2573,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2158,6 +2591,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2169,19 +2603,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2210,6 +2653,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2285,6 +2729,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2403,6 +2867,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2422,15 +2918,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2723,6 +3250,498 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 19c7bac..7d08e2f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
0008-Support-logical_decoding_work_mem-set-from-create-su.patch (application/octet-stream, 13.1K) download
From 54556711ea38bcd407bdc65f12f6c70ec2e8a592 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH 08/13] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d4d1fe4..f449fa1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1753,6 +1753,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff..2a27648 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edc..fbb4473 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fc..0ab6855 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ff62303..14c0ce8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1726,6 +1726,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b..cf6e03b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934..4e68a69 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
0010-Track-statistics-for-streaming.patch (application/octet-stream, 11.6K) download
From bf0e4ba0c0441cbbf52989f35b49fdbf7da01fdc Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 12:19:49 +0530
Subject: [PATCH 10/13] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++++
src/backend/catalog/system_views.sql | 5 ++++-
src/backend/replication/logical/reorderbuffer.c | 13 +++++++++++
src/backend/replication/walsender.c | 30 ++++++++++++++++++++-----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 +++++++----
src/include/replication/walsender_private.h | 5 +++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 89 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index a3c5f86..3de62c0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1992,6 +1992,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f0..5897611 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -779,7 +779,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0ab3191..83eb4df 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -358,6 +358,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3732,6 +3736,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
PG_END_TRY();
/*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
+ /*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a2ae283..8619a01 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1269,7 +1269,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1290,7 +1290,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2334,6 +2335,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3235,7 +3239,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3293,6 +3297,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3316,6 +3323,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3402,6 +3412,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3650,8 +3665,13 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld",
- rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes,
+ rb->streamTxns, rb->streamCount, rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fa0a2a1..9a508bf 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d08e2f..c183fce 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -511,15 +511,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index a6b3205..7efc332 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index c9cc569..9fe3dd5 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1955,9 +1955,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
0009-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 89.7K) download
From 81193e00db98be3807352d84e712d13b5b4049d9 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:53:58 +0530
Subject: [PATCH 09/13] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 2a27648..15a6f5a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fbb4473..b2b93d6 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index fabcf31..75effed 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4104,6 +4104,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0ab6855..9970170 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 1f8821c..9307b67 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b88b585..ad43ab3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47d..5a379fb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 14c0ce8..65e47aa 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -59,6 +81,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -66,6 +89,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -105,6 +129,50 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
@@ -114,6 +182,9 @@ static void maybe_reread_subscription(void);
/* Flags set by signal handlers */
static volatile sig_atomic_t got_SIGHUP = false;
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -165,6 +236,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -512,6 +619,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -524,6 +943,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -539,6 +961,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -574,6 +999,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -677,6 +1105,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -797,6 +1228,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -896,6 +1330,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -987,6 +1424,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1084,6 +1537,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1099,6 +1568,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1547,6 +2019,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* SIGHUP: set flag to reload configuration at next convenient time */
static void
logicalrep_worker_sighup(SIGNAL_ARGS)
@@ -1727,6 +2757,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index cf6e03b..8490ea4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,16 +45,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -64,6 +90,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -84,16 +111,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +199,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +228,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +252,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +273,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +361,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +422,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +450,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +482,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +499,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +508,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +534,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +558,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +639,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -623,6 +834,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
+/*
* Relcache invalidation callback
*/
static void
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 46e6dd4..c98d476 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index fa75872..a2ae283 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -945,6 +945,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113..8793676 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fe076d8..bc45194 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -944,7 +944,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430a..bf02cbc 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 4e68a69..fe6acb4 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
0012-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1010B) download
From 9699ab41ef32af4a0bafff7d4263a9d6d534ccc4 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH 12/13] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 83eb4df..410da36 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
0011-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From cf6ce6192de8ebcaa8942fb234426e30e976055d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH 11/13] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 40e306a..f41a0e1 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -64,7 +64,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f6..8dfeafc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
0013-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 79bba6bd97b938917720012d99a19ec614752976 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH 13/13] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-11-21 03:32 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-21 03:32 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Nov 20, 2019 at 8:22 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Nov 20, 2019 at 11:15 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Nov 19, 2019 at 5:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Mon, Nov 18, 2019 at 5:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > > >
> > > > > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > > >
> > > > > > >
> > > > > > > Few other comments on this patch:
> > > > > > > 1.
> > > > > > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > > > > > +
> > > > > > > + /*
> > > > > > > + * Execute the invalidation message locally.
> > > > > > > + *
> > > > > > > + * XXX Do we need to care about relcacheInitFileInval and
> > > > > > > + * the other fields added to ReorderBufferChange, or just
> > > > > > > + * about the message itself?
> > > > > > > + */
> > > > > > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > > > > > + break;
> > > > > > >
> > > > > > > Here, why are we executing messages individually? Can't we just
> > > > > > > follow what we do in DecodeCommit which is to record the invalidations
> > > > > > > in ReorderBufferTXN as we encounter them and then allow them to
> > > > > > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > > > > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > > > > > receive any invalidation message?
> > > >
> > > > I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
> > > > commit. Because this is required to add any committed transaction to
> > > > the snapshot if it has done any catalog changes.
> > > >
> > >
> > > Hmm, this is also used to build cid hash map (see
> > > ReorderBufferBuildTupleCidHash) which we need to use while streaming
> > > changes for the in-progress transactions. So, I think that it would
> > > be required earlier (before commit) as well.
> > >
> > Oh right, I guess I missed that part.
>
> Attached a new rebased version of the patch set. I have fixed all
> the issues discussed up-thread and agreed upon.
>
> Pending Issues:
> 1. The default value of the logical_decoding_work_mem is set to 64kb
> in test_decoding/logical.conf. So we need to change the expected
> output files for the test decoding module.
> 2. Need to complete the patch for concurrent abort handling of the
> (sub)transaction. There are some pending issues with the existing
> patch[1].
> [1] https://www.postgresql.org/message-id/CAFiTN-ud98kWHCo2YKS55H8rGw3_A7ESyssHwU0xPU6KJsoy6A%40mail.gma...
Apart from these there is one more issue reported upthread[2]
[2] https://www.postgresql.org/message-id/CAFiTN-vrSNkAfRVrWKe2R1dqFBTubjt%3DDYS%3DjhH%2BjiCoBODdaw%40ma...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-11-22 07:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-11-22 07:48 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Nov 21, 2019 at 9:02 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Nov 20, 2019 at 8:22 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Wed, Nov 20, 2019 at 11:15 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Tue, Nov 19, 2019 at 5:23 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Mon, Nov 18, 2019 at 5:02 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > >
> > > > > On Fri, Nov 15, 2019 at 4:19 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > >
> > > > > > On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > > > >
> > > > > > > On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > Few other comments on this patch:
> > > > > > > > 1.
> > > > > > > > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > > > > > > > +
> > > > > > > > + /*
> > > > > > > > + * Execute the invalidation message locally.
> > > > > > > > + *
> > > > > > > > + * XXX Do we need to care about relcacheInitFileInval and
> > > > > > > > + * the other fields added to ReorderBufferChange, or just
> > > > > > > > + * about the message itself?
> > > > > > > > + */
> > > > > > > > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > > > > > > > + break;
> > > > > > > >
> > > > > > > > Here, why are we executing messages individually? Can't we just
> > > > > > > > follow what we do in DecodeCommit which is to record the invalidations
> > > > > > > > in ReorderBufferTXN as we encounter them and then allow them to
> > > > > > > > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > > > > > > > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > > > > > > > receive any invalidation message?
> > > > >
> > > > > I think it's fine to call ReorderBufferXidSetCatalogChanges, only on
> > > > > commit. Because this is required to add any committed transaction to
> > > > > the snapshot if it has done any catalog changes.
> > > > >
> > > >
> > > > Hmm, this is also used to build cid hash map (see
> > > > ReorderBufferBuildTupleCidHash) which we need to use while streaming
> > > > changes for the in-progress transactions. So, I think that it would
> > > > be required earlier (before commit) as well.
> > > >
> > > Oh right, I guess I missed that part.
> >
> > Attached a new rebased version of the patch set. I have fixed all
> > the issues discussed up-thread and agreed upon.
> >
> > Pending Issues:
> > 1. The default value of the logical_decoding_work_mem is set to 64kb
> > in test_decoding/logical.conf. So we need to change the expected
> > output files for the test decoding module.
> > 2. Need to complete the patch for concurrent abort handling of the
> > (sub)transaction. There are some pending issues with the existing
> > patch[1].
> > [1] https://www.postgresql.org/message-id/CAFiTN-ud98kWHCo2YKS55H8rGw3_A7ESyssHwU0xPU6KJsoy6A%40mail.gma...
> Apart from these there is one more issue reported upthread[2]
> [2] https://www.postgresql.org/message-id/CAFiTN-vrSNkAfRVrWKe2R1dqFBTubjt%3DDYS%3DjhH%2BjiCoBODdaw%40ma...
>
I have rebased the patch on the latest head and also fix the issue of
"concurrent abort handling of the (sub)transaction." and attached as
(v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
the complete patch set. I have added the version number so that we
can track the changes.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v1-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.2K) download
From 949e0782796eef008afd703c85a8eaca6625709d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v1 01/13] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So instead we write the assignment info into WAL immediately, as
part of the next WAL record (to minimize overhead).
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 8fe38c3..4a853f3 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index aa9dca0..a8a8084 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 7f24f0c..4ef2661 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1071,6 +1071,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1109,6 +1110,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index bc532d0..897b755 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 9d2899d..5b9740c 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252..b492d3e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 1bbee38..c37a83d 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -148,6 +148,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -243,6 +245,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54..bcfba0a 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v1-0002-Issue-individual-invalidations-with-wal_level-log.patch (application/octet-stream, 15.8K) download
From 4ddd0a234ec138105302d62bb9900a8c858a225e Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v1 02/13] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 52 +++++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 +++++++++
src/backend/replication/logical/reorderbuffer.c | 56 +++++++++++++++++---
src/backend/utils/cache/inval.c | 69 +++++++++++++++++++++++++
src/include/access/xact.h | 18 ++++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 4c411c5..6cfd6af 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,46 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 4a853f3..a42d11f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 897b755..9bcefb6 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 53affeb..b1feff3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -464,6 +464,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1804,17 +1805,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2209,6 +2216,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2656,6 +2695,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2752,6 +2792,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3027,6 +3068,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9..0682c55 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1546,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 5b9740c..82d4942 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0867ee9..6a7187b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v1-0004-Cleaning-up-of-flags-in-ReorderBufferTXN-structur.patch (application/octet-stream, 8.1K) download
From ad5a84eef45bc215044748f382e0d5d586a502be Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH v1 04/13] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++---------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b1feff3..3422939 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -732,7 +732,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -752,7 +752,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -775,7 +775,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -835,7 +835,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -851,7 +851,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1061,7 +1061,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1090,7 +1090,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1256,7 +1256,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1304,7 +1304,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1319,7 +1319,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1336,7 +1336,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1970,7 +1970,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2118,7 +2118,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2297,7 +2297,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2314,7 +2314,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2334,7 +2334,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2522,12 +2522,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b4be2b..19c7bac 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
v1-0005-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (application/octet-stream, 12.5K) download
From 193f2812afa906a07cc5f90a8b872cebbf96402b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v1 05/13] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 9 +++--
src/backend/utils/time/snapmgr.c | 25 +++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 120 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65..da6a6f3 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0128bb3..2a60a73 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1433,16 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1714,16 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3422939..bda4a1c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -683,7 +683,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1533,7 +1533,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1784,7 +1784,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1804,7 +1804,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
@@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v1-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.9K) download
From d3e02ec7f0485638247e236937e25d62a1f44ab3 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v1 03/13] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4b..9c77791 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..fc4ad65 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting. At
+ that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 7e06615..b88b585 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 31c796b..d95d1b9 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -81,6 +81,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f..a305462 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6a7187b..5b4be2b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v1-0006-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 44.3K) download
From 95cc2862df84b5bc02ca7ac0a3ab8bd2f95f52f2 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:42:31 +0530
Subject: [PATCH v1 06/13] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 1075 ++++++++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1112 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 3e36467..cf10dd0 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index bda4a1c..0ab3191 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -362,6 +408,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -759,6 +808,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -855,6 +931,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -978,7 +1057,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1006,6 +1085,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1020,6 +1102,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1235,6 +1320,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
+/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1327,33 +1616,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1403,6 +1763,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1476,6 +1846,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1515,6 +1898,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_lsn = origin_lsn;
/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
* ReorderBufferCommitChild will have transferred any snapshots from
@@ -1549,6 +1948,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1930,6 +2340,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2014,6 +2431,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2149,8 +2573,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2158,6 +2591,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2169,19 +2603,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2210,6 +2653,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2285,6 +2729,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2403,6 +2867,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2422,15 +2918,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2723,6 +3250,498 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 19c7bac..7d08e2f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v1-0007-Support-logical_decoding_work_mem-set-from-create.patch (application/octet-stream, 13.1K) download
From ac49838b249e181e0fbd9297b034b96b6636f964 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v1 07/13] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d4d1fe4..f449fa1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1753,6 +1753,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff..2a27648 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edc..fbb4473 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fc..0ab6855 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ff62303..14c0ce8 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1726,6 +1726,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b..cf6e03b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934..4e68a69 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
v1-0008-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 89.7K) download
From 3968b2aa385b703316109df3eaef0dce7e027f0b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:53:58 +0530
Subject: [PATCH v1 08/13] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 2a27648..15a6f5a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fbb4473..b2b93d6 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index fabcf31..75effed 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4104,6 +4104,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0ab6855..9970170 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 1f8821c..9307b67 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b88b585..ad43ab3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47d..5a379fb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 14c0ce8..65e47aa 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -59,6 +81,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -66,6 +89,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -105,6 +129,50 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
@@ -114,6 +182,9 @@ static void maybe_reread_subscription(void);
/* Flags set by signal handlers */
static volatile sig_atomic_t got_SIGHUP = false;
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -165,6 +236,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -512,6 +619,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -524,6 +943,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -539,6 +961,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -574,6 +999,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -677,6 +1105,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -797,6 +1228,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -896,6 +1330,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -987,6 +1424,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1084,6 +1537,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1099,6 +1568,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1547,6 +2019,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* SIGHUP: set flag to reload configuration at next convenient time */
static void
logicalrep_worker_sighup(SIGNAL_ARGS)
@@ -1727,6 +2757,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index cf6e03b..8490ea4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,16 +45,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -64,6 +90,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -84,16 +111,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +199,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +228,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +252,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +273,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +361,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +422,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +450,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +482,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +499,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +508,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +534,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +558,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +639,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -623,6 +834,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
+/*
* Relcache invalidation callback
*/
static void
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 46e6dd4..c98d476 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index fa75872..a2ae283 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -945,6 +945,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113..8793676 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fe076d8..bc45194 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -944,7 +944,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430a..bf02cbc 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 4e68a69..fe6acb4 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v1-0009-Track-statistics-for-streaming.patch (application/octet-stream, 11.6K) download
From 9ebaada8dc1630ba7b039da109f14f3eb111ca50 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 12:19:49 +0530
Subject: [PATCH v1 09/13] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++++
src/backend/catalog/system_views.sql | 5 ++++-
src/backend/replication/logical/reorderbuffer.c | 13 +++++++++++
src/backend/replication/walsender.c | 30 ++++++++++++++++++++-----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 +++++++----
src/include/replication/walsender_private.h | 5 +++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 89 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index a3c5f86..3de62c0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1992,6 +1992,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f0..5897611 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -779,7 +779,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0ab3191..83eb4df 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -358,6 +358,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3732,6 +3736,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
PG_END_TRY();
/*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
+ /*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index a2ae283..8619a01 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1269,7 +1269,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1290,7 +1290,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2334,6 +2335,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3235,7 +3239,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3293,6 +3297,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3316,6 +3323,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3402,6 +3412,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3650,8 +3665,13 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld",
- rb, rb->spillTxns, rb->spillCount, rb->spillBytes);
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %ld %ld %ld %ld %ld %ld",
+ rb, rb->spillTxns, rb->spillCount, rb->spillBytes,
+ rb->streamTxns, rb->streamCount, rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index d4fa6dd..b3bbc82 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d08e2f..c183fce 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -511,15 +511,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index a6b3205..7efc332 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index c9cc569..9fe3dd5 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1955,9 +1955,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v1-0010-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From 989a8c4e3f0574949f36e2ecaa685548aa7726b1 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v1 10/13] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 40e306a..f41a0e1 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -64,7 +64,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f6..8dfeafc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v1-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1013B) download
From 5bfee24870b38fb1b60b1c0806f98fd289edabc6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v1 11/13] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 83eb4df..410da36 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
v1-0012-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 739e742dfe7b453857d6ef4d354cfbfe1d25c111 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v1 12/13] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v1-0013-Extend-handling-of-concurrent-aborts-for-streamin.patch (application/octet-stream, 4.0K) download
From 39df08b4bbe216f2fe1a386c5afd63775565099a Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 22 Nov 2019 12:43:38 +0530
Subject: [PATCH v1 13/13] Extend handling of concurrent aborts for streaming
transaction
---
src/backend/replication/logical/reorderbuffer.c | 38 +++++++++++++++++++++++--
src/include/replication/reorderbuffer.h | 5 ++++
2 files changed, 40 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 410da36..710a22f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2350,9 +2350,9 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
/*
* When the (sub)transaction was streamed, notify the remote node
- * about the abort.
+ * about the abort only if we have sent any data for this transaction.
*/
- if (rbtxn_is_streamed(txn))
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
rb->stream_abort(rb, txn, lsn);
/* cosmetic... */
@@ -3281,6 +3281,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3411,6 +3412,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3472,6 +3480,10 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferToastReplace(rb, txn, relation, change);
rb->stream_change(rb, txn, relation, change);
+ /* Remember that we have sent some data for this txn.*/
+ if (!change->txn->any_data_sent)
+ change->txn->any_data_sent = true;
+
/*
* Only clear reassembled toast chunks if we're sure
* they're not required anymore. The creator of the
@@ -3654,6 +3666,8 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
+
+ /* Invalidate current schema as well */
txn->is_schema_sent = false;
break;
@@ -3717,6 +3731,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3735,7 +3752,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
}
PG_END_TRY();
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index c183fce..1db6da6 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -242,6 +242,11 @@ typedef struct ReorderBufferTXN
bool is_schema_sent;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
* Toplevel transaction for this subxact (NULL for top-level).
*/
struct ReorderBufferTXN *toptxn;
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-12-01 02:28 ` Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Michael Paquier @ 2019-12-01 02:28 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> I have rebased the patch on the latest head and also fix the issue of
> "concurrent abort handling of the (sub)transaction." and attached as
> (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> the complete patch set. I have added the version number so that we
> can track the changes.
The patch has rotten a bit and does not apply anymore. Could you
please send a rebased version? I have moved it to next CF, waiting on
author.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
@ 2019-12-02 08:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-10 04:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-11 18:16 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Robert Haas <robertmhaas@gmail.com>
2019-12-20 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
2019-12-22 11:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions vignesh C <vignesh21@gmail.com>
0 siblings, 5 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-12-02 08:31 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > I have rebased the patch on the latest head and also fix the issue of
> > "concurrent abort handling of the (sub)transaction." and attached as
> > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > the complete patch set. I have added the version number so that we
> > can track the changes.
>
> The patch has rotten a bit and does not apply anymore. Could you
> please send a rebased version? I have moved it to next CF, waiting on
> author.
I have rebased the patch set on the latest head.
Apart from this, there is one issue reported by my colleague Vignesh.
The issue is that if we use more than two relations in a transaction
then there is an error on standby (no relation map entry for remote
relation ID 16390). After analyzing I have found that for the
streaming transaction an "is_schema_sent" flag is kept in
ReorderBufferTXN. And, I think that is done so that we can send the
schema for each transaction stream so that if any subtransaction gets
aborted we don't lose the logical WAL for that schema. But, this
solution has induced a very basic issue that if a transaction operate
on more than 1 relation then after sending the schema for the first
relation it will mark the flag true and the schema for the subsequent
relations will never be sent. I am still working on finding a better
solution for this if anyone has any opinion/solution about this feel
free to suggest.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v2-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.2K) download
From 05fe8c63cc2e81b32aaf72599190fa4521362429 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v2 01/13] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So instead we write the assignment info into WAL immediately, as
part of the next WAL record (to minimize overhead).
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 5353b6a..708e523 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index aa9dca0..a8a8084 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 67418b0..4435c63 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index bc532d0..897b755 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 9d2899d..5b9740c 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252..b492d3e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 0193611..a676151 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54..bcfba0a 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v2-0002-Issue-individual-invalidations-with-wal_level-log.patch (application/octet-stream, 15.8K) download
From a10bea688773350e4262839d9d64df2ec2b7c429 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v2 02/13] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 52 +++++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 +++++++++
src/backend/replication/logical/reorderbuffer.c | 56 +++++++++++++++++---
src/backend/utils/cache/inval.c | 69 +++++++++++++++++++++++++
src/include/access/xact.h | 18 ++++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 231 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 4c411c5..6cfd6af 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,46 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 708e523..da15556 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 897b755..9bcefb6 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 53affeb..b1feff3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -464,6 +464,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1804,17 +1805,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2209,6 +2216,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2656,6 +2695,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2752,6 +2792,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3027,6 +3068,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9..0682c55 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1546,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 5b9740c..82d4942 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0867ee9..6a7187b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v2-0004-Cleaning-up-of-flags-in-ReorderBufferTXN-structur.patch (application/octet-stream, 8.1K) download
From b6825f74dfd9809dc06feb16073eab5714a4a488 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH v2 04/13] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++---------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b1feff3..3422939 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -732,7 +732,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -752,7 +752,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -775,7 +775,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -835,7 +835,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -851,7 +851,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1061,7 +1061,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1090,7 +1090,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1256,7 +1256,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1304,7 +1304,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1319,7 +1319,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1336,7 +1336,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1970,7 +1970,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2118,7 +2118,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2297,7 +2297,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2314,7 +2314,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2334,7 +2334,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2522,12 +2522,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b4be2b..19c7bac 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
v2-0005-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (application/octet-stream, 12.5K) download
From 2b3a261b2c80aa84073c39d7be76a42d6e491195 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v2 05/13] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions - for example when decoding prepared
transactions on PREPARE (and not COMMIT PREPARED as before), this
may cause failures when the output plugin consults catalogs (both
system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 51 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 +++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 9 +++--
src/backend/utils/time/snapmgr.c | 25 +++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 120 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index fc4ad65..da6a6f3 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0128bb3..2a60a73 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,17 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_getnext call")));
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1433,16 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_fetch call")));
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1556,16 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search_buffer call")));
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1714,16 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_get_latest_tid call")));
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5522,16 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_TRANSACTION_STATE),
+ errmsg("improper heap_hot_search call")));
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3422939..bda4a1c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -683,7 +683,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1533,7 +1533,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1784,7 +1784,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1804,7 +1804,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
@@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v2-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.9K) download
From 64fa40c143ef89e61590a45f93f8e3dc675af4f2 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v2 03/13] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4b..9c77791 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..fc4ad65 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting. At
+ that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 7e06615..b88b585 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ /* FIXME ctx->write_location = apply_lsn; */
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 6879a2e..1e934d2 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f..a305462 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6a7187b..5b4be2b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v2-0006-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 44.3K) download
From bd34ab5db8501a4a8ea06bcd53c4386fe660a7ae Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:42:31 +0530
Subject: [PATCH v2 06/13] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 1075 ++++++++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1112 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 3e36467..cf10dd0 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index bda4a1c..0ab3191 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -362,6 +408,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -759,6 +808,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -855,6 +931,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -978,7 +1057,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1006,6 +1085,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1020,6 +1102,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1235,6 +1320,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
+/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1327,33 +1616,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1403,6 +1763,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1476,6 +1846,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1515,6 +1898,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_lsn = origin_lsn;
/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
* ReorderBufferCommitChild will have transferred any snapshots from
@@ -1549,6 +1948,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1930,6 +2340,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2014,6 +2431,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2149,8 +2573,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2158,6 +2591,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2169,19 +2603,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2210,6 +2653,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2285,6 +2729,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2403,6 +2867,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2422,15 +2918,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2723,6 +3250,498 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 19c7bac..7d08e2f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v2-0007-Support-logical_decoding_work_mem-set-from-create.patch (application/octet-stream, 13.1K) download
From 8f7065a4c1347fcf27b89650ab4a6cc6e9d6d012 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v2 07/13] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4ec13f3..3c32686 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1753,6 +1753,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff..2a27648 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edc..fbb4473 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fc..0ab6855 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ced0d19..ebb976c 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1742,6 +1742,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b..cf6e03b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 41714ea..1db706a 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
v2-0008-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 89.7K) download
From 6aa8ffd27af999e878f1fd53f83d3a70dfbec0ac Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:53:58 +0530
Subject: [PATCH v2 08/13] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 2a27648..15a6f5a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fbb4473..b2b93d6 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index fabcf31..75effed 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4104,6 +4104,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0ab6855..9970170 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index 4643af9..15e7140 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index b88b585..ad43ab3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47d..5a379fb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ebb976c..62be080 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -59,6 +81,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -66,6 +89,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -105,6 +129,50 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
@@ -114,6 +182,9 @@ static void maybe_reread_subscription(void);
/* Flags set by signal handlers */
static volatile sig_atomic_t got_SIGHUP = false;
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -165,6 +236,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -528,6 +635,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -540,6 +959,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -555,6 +977,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -590,6 +1015,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -693,6 +1121,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -813,6 +1244,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -912,6 +1346,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1003,6 +1440,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1100,6 +1553,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1115,6 +1584,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1563,6 +2035,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* SIGHUP: set flag to reload configuration at next convenient time */
static void
logicalrep_worker_sighup(SIGNAL_ARGS)
@@ -1743,6 +2773,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index cf6e03b..8490ea4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,16 +45,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -64,6 +90,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -84,16 +111,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +199,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +228,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +252,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +273,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +361,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +422,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +450,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +482,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +499,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +508,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +534,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +558,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +639,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -623,6 +834,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
+/*
* Relcache invalidation callback
*/
static void
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 46e6dd4..c98d476 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 8bafa65..0837264 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -968,6 +968,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113..8793676 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index fe076d8..bc45194 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -944,7 +944,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430a..bf02cbc 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 1db706a..3d19b5d 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v2-0009-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From ebc31a6153b7a4e7887d5d8b7c6c4959c54f4e60 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 2 Dec 2019 09:58:50 +0530
Subject: [PATCH v2 09/13] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index a3c5f86..3de62c0 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1992,6 +1992,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f0..5897611 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -779,7 +779,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0ab3191..83eb4df 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -358,6 +358,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3732,6 +3736,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
PG_END_TRY();
/*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
+ /*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 0837264..9f34b3b 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1292,7 +1292,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1313,7 +1313,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2356,6 +2357,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3178,7 +3182,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3236,6 +3240,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3259,6 +3266,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3345,6 +3355,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3592,12 +3607,19 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillTxns = rb->spillTxns;
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
+
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ac8f64b..3b897a5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d08e2f..c183fce 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -511,15 +511,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index a6b3205..7efc332 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index c9cc569..9fe3dd5 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1955,9 +1955,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v2-0010-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From a97bae3b74b77b77df1ca134f61a764e6cea1bd3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v2 10/13] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 77a1560..8cd1993 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -65,7 +65,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f6..8dfeafc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v2-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1013B) download
From e468e357589eb43acb2b1117f9b3ada7965c86e1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v2 11/13] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 83eb4df..410da36 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
v2-0012-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 24b2c427831ba1e268f834ea368f796b38a81ad1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v2 12/13] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v2-0013-Extend-handling-of-concurrent-aborts-for-streamin.patch (application/octet-stream, 4.0K) download
From a0b6c483c0180771e9fd0a21f13aa3ce924fb0db Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 22 Nov 2019 12:43:38 +0530
Subject: [PATCH v2 13/13] Extend handling of concurrent aborts for streaming
transaction
---
src/backend/replication/logical/reorderbuffer.c | 38 +++++++++++++++++++++++--
src/include/replication/reorderbuffer.h | 5 ++++
2 files changed, 40 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 410da36..710a22f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2350,9 +2350,9 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
/*
* When the (sub)transaction was streamed, notify the remote node
- * about the abort.
+ * about the abort only if we have sent any data for this transaction.
*/
- if (rbtxn_is_streamed(txn))
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
rb->stream_abort(rb, txn, lsn);
/* cosmetic... */
@@ -3281,6 +3281,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3411,6 +3412,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3472,6 +3480,10 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferToastReplace(rb, txn, relation, change);
rb->stream_change(rb, txn, relation, change);
+ /* Remember that we have sent some data for this txn.*/
+ if (!change->txn->any_data_sent)
+ change->txn->any_data_sent = true;
+
/*
* Only clear reassembled toast chunks if we're sure
* they're not required anymore. The creator of the
@@ -3654,6 +3666,8 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
+
+ /* Invalidate current schema as well */
txn->is_schema_sent = false;
break;
@@ -3717,6 +3731,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3735,7 +3752,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
}
PG_END_TRY();
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index c183fce..1db6da6 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -242,6 +242,11 @@ typedef struct ReorderBufferTXN
bool is_schema_sent;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
* Toplevel transaction for this subxact (NULL for top-level).
*/
struct ReorderBufferTXN *toptxn;
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-12-09 07:56 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
4 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-12-09 07:56 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 2, 2019 at 2:01 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:
> >
> > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > > I have rebased the patch on the latest head and also fix the issue of
> > > "concurrent abort handling of the (sub)transaction." and attached as
> > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > > the complete patch set. I have added the version number so that we
> > > can track the changes.
> >
> > The patch has rotten a bit and does not apply anymore. Could you
> > please send a rebased version? I have moved it to next CF, waiting on
> > author.
>
> I have rebased the patch set on the latest head.
>
I have review the patch set and here are few comments/questions
1.
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
Should we show the tuple in the streamed change like we do for the
pg_decode_change?
2. pg_logical_slot_get_changes_guts
It recreate the decoding slot [ctx =
CreateDecodingContext(InvalidXLogRecPtr] but doesn't set the streaming
to false, should we pass a parameter to
pg_logical_slot_get_changes_guts saying whether we want streamed results or not
3.
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
I did not understand, how this change is relavent to this patch
4.
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
In which case, txn->is_schema_sent will be true, because at the end of
the stream in ReorderBufferExecuteInvalidations we are always setting
it false,
so while sending next stream it will always be false. That means we
never required snapshot_now variable in ReorderBufferTXN.
5.
@@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer
*rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
Same as point 4, during decode time it will never be true.
6.
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
Commit_time and end_lsn is used in standby_feedback
7.
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
We can not rollback intermediate subtransaction without rollbacking
latest sub-transaction, so why do we need
to search in the array? It will always be the the last subxact no?
8.
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
Why feedback is sent for every change?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-12-11 11:52 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2019-12-11 11:52 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 9, 2019 at 1:27 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have review the patch set and here are few comments/questions
>
> 1.
> +static void
> +pg_decode_stream_change(LogicalDecodingContext *ctx,
> + ReorderBufferTXN *txn,
> + Relation relation,
> + ReorderBufferChange *change)
> +{
> + OutputPluginPrepareWrite(ctx, true);
> + appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
> + OutputPluginWrite(ctx, true);
> +}
>
> Should we show the tuple in the streamed change like we do for the
> pg_decode_change?
>
I think so. The patch shows the message in
pg_decode_stream_message(), so why to prohibit showing tuple here?
> 2. pg_logical_slot_get_changes_guts
> It recreate the decoding slot [ctx =
> CreateDecodingContext(InvalidXLogRecPtr] but doesn't set the streaming
> to false, should we pass a parameter to
> pg_logical_slot_get_changes_guts saying whether we want streamed results or not
>
CreateDecodingContext internally calls StartupDecodingContext which
sets the value of streaming based on if the plugin has provided
callbacks for streaming functions. Isn't that sufficient? Why do we
need additional parameters here?
> 3.
> + XLogRecPtr prev_lsn = InvalidXLogRecPtr;
> ReorderBufferChange *change;
> ReorderBufferChange *specinsert = NULL;
>
> @@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> Relation relation = NULL;
> Oid reloid;
>
> + /*
> + * Enforce correct ordering of changes, merged from multiple
> + * subtransactions. The changes may have the same LSN due to
> + * MULTI_INSERT xlog records.
> + */
> + if (prev_lsn != InvalidXLogRecPtr)
> + Assert(prev_lsn <= change->lsn);
> +
> + prev_lsn = change->lsn;
> I did not understand, how this change is relavent to this patch
>
This is just to ensure that changes are in LSN order. I think as we
are merging the changes before commit for streaming, it is good to
have such an Assertion for ReorderBufferStreamTXN. And, if we want
to have it in ReorderBufferStreamTXN, then there is no harm in keeping
it in ReorderBufferCommit() at least to keep the code consistent. Do
you see any problem with this?
> 4.
> + /*
> + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> + * information about subtransactions, which could arrive after streaming start.
> + */
> + if (!txn->is_schema_sent)
> + snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
> + txn, command_id);
>
> In which case, txn->is_schema_sent will be true, because at the end of
> the stream in ReorderBufferExecuteInvalidations we are always setting
> it false,
> so while sending next stream it will always be false. That means we
> never required snapshot_now variable in ReorderBufferTXN.
>
You are probably right, but as discussed we need to change this part
of design/code (when to send schema changes) due to the issues
discovered. So, I think this part will anyway change when we fix that
problem.
> 5.
> @@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer
> *rb, TransactionId xid,
> txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
>
> txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
> +
> + /*
> + * We read catalog changes from WAL, which are not yet sent, so
> + * invalidate current schema in order output plugin can resend
> + * schema again.
> + */
> + txn->is_schema_sent = false;
>
> Same as point 4, during decode time it will never be true.
>
Sure, my previous point's reply applies here as well.
> 6.
> + /* send fields */
> + pq_sendint64(out, commit_lsn);
> + pq_sendint64(out, txn->end_lsn);
> + pq_sendint64(out, txn->commit_time);
>
> Commit_time and end_lsn is used in standby_feedback
>
I don't understand what you mean by this. Can you be a bit more clear?
>
> 7.
> + /* FIXME optimize the search by bsearch on sorted data */
> + for (i = nsubxacts; i > 0; i--)
> + {
> + if (subxacts[i - 1].xid == subxid)
> + {
> + subidx = (i - 1);
> + found = true;
> + break;
> + }
> + }
> We can not rollback intermediate subtransaction without rollbacking
> latest sub-transaction, so why do we need
> to search in the array? It will always be the the last subxact no?
>
The same thing is already mentioned in the comments above this code
("XXX Or perhaps we can rely on the aborts to arrive in the reverse
order, i.e. from the inner-most subxact (when nested)? In which case
we could simply check the last element."). I think what you are
saying is probably right, but we can leave this as it is for now
because this is a minor optimization which can be done later as well
if required. However, if you see any correctness issue, then we can
discuss.
> 8.
> + /*
> + * send feedback to upstream
> + *
> + * XXX Probably should send a valid LSN. But which one?
> + */
> + send_feedback(InvalidXLogRecPtr, false, false);
>
> Why feedback is sent for every change?
>
I will study this part of the patch and let you know my opinion.
Few comments on this patch series:
0001-Immediately-WAL-log-assignments:
------------------------------------------------------------
The commit message still refers to the old design for this patch. I
think you need to modify the commit message as per the latest patch.
0002-Issue-individual-invalidations-with-wal_level-log
----------------------------------------------------------------------------
1.
xact_desc_invalidations(StringInfo buf,
{
..
+ else if (msg->id == SHAREDINVALSNAPSHOT_ID)
+ appendStringInfo(buf, " snapshot %u", msg->sn.relId);
You have removed logging for the above cache but forgot to remove its
reference from one of the places. Also, I think you need to add a
comment somewhere in inval.c to say why you are writing for WAL for
some types of invalidations and not for others?
0003-Extend-the-output-plugin-API-with-stream-methods
--------------------------------------------------------------------------------
1.
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_message_cb</function> are optional.
stream_message_cb is mentioned twice. It seems the second one is for truncate.
2.
size of the transaction size and network bandwidth, the transfer time
+ may significantly increase the apply lag.
/size of the transaction size/size of the transaction
no need to mention size twice.
3.
+ Similarly to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress
transactions)
+ exceeds limit defined by <varname>logical_work_mem</varname> setting.
The guc name used is wrong. /Similarly to/Similar to/
4.
stream_start_cb_wrapper()
{
..
+ /* state.report_location = apply_lsn; */
..
+ /* FIXME ctx->write_location = apply_lsn; */
..
}
See, if we can fix these and similar in the callback for the stop. I
think we don't have final_lsn till we commit/abort. Can we compute
before calling these API's?
0005-Gracefully-handle-concurrent-aborts-of-uncommitte
----------------------------------------------------------------------------------
1.
@@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_CATCH();
{
/* TODO: Encapsulate cleanup
from the PG_TRY and PG_CATCH blocks */
+
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
Spurious line change.
2. The commit message of this patch refers to Prepared transactions.
I think that needs to be changed.
0006-Implement-streaming-mode-in-ReorderBuffer
-------------------------------------------------------------------------
1.
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+
ReorderBuffer *rb,
+
ReorderBufferTXN
*txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+
ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+
ReorderBuffer *rb,
+
ReorderBufferStreamIterTXNState * state);
Do we really need to introduce new APIs for iterating over changes
from streamed transactions? Why can't we reuse the same API's as we
use for committed xacts?
2.
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
Please write some comments atop ReorderBufferStreamCommit.
3.
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
..
..
+ if (txn->snapshot_now
== NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+
Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id ==
InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+
subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+
(txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+
txn->base_snapshot = subtxn->base_snapshot;
The logic here seems to be correct, but I am not sure why it is not
considered to purge the base snapshot before assigning the subtxn's
snapshot and similarly, we have not purged snapshot for subtxn once we
are done with it. I think we can use
ReorderBufferTransferSnapToParent to replace part of the logic here.
Do you see any reason for doing things differently here?
4. In ReorderBufferStreamTXN, why do you need to use
ReorderBufferCopySnap to assign txn->base_snapshot to snapshot_now.
5. I see a lot of code similarity in ReorderBufferStreamTXN and
existing ReorderBufferCommit. I understand that there are some subtle
differences due to which we need to write this new function but can't
we encapsulate the specific parts of code in functions and then call
from both places. I am talking about code in different cases for
change->action.
6. + * Note: We never stream and serialize a transaction at the same time (e
/(e/(we
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2019-12-12 04:14 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-13 10:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2019-12-12 04:14 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Dec 11, 2019 at 5:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Dec 9, 2019 at 1:27 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have review the patch set and here are few comments/questions
> >
> > 1.
> > +static void
> > +pg_decode_stream_change(LogicalDecodingContext *ctx,
> > + ReorderBufferTXN *txn,
> > + Relation relation,
> > + ReorderBufferChange *change)
> > +{
> > + OutputPluginPrepareWrite(ctx, true);
> > + appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
> > + OutputPluginWrite(ctx, true);
> > +}
> >
> > Should we show the tuple in the streamed change like we do for the
> > pg_decode_change?
> >
>
> I think so. The patch shows the message in
> pg_decode_stream_message(), so why to prohibit showing tuple here?
>
> > 2. pg_logical_slot_get_changes_guts
> > It recreate the decoding slot [ctx =
> > CreateDecodingContext(InvalidXLogRecPtr] but doesn't set the streaming
> > to false, should we pass a parameter to
> > pg_logical_slot_get_changes_guts saying whether we want streamed results or not
> >
>
> CreateDecodingContext internally calls StartupDecodingContext which
> sets the value of streaming based on if the plugin has provided
> callbacks for streaming functions. Isn't that sufficient? Why do we
> need additional parameters here?
I don't think that if plugin provides streaming function then we
should stream. Like pgoutput plugin provides streaming function but
we only stream if streaming is on in create subscription command. So
I feel that should be true with any plugin.
>
> > 3.
> > + XLogRecPtr prev_lsn = InvalidXLogRecPtr;
> > ReorderBufferChange *change;
> > ReorderBufferChange *specinsert = NULL;
> >
> > @@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> > Relation relation = NULL;
> > Oid reloid;
> >
> > + /*
> > + * Enforce correct ordering of changes, merged from multiple
> > + * subtransactions. The changes may have the same LSN due to
> > + * MULTI_INSERT xlog records.
> > + */
> > + if (prev_lsn != InvalidXLogRecPtr)
> > + Assert(prev_lsn <= change->lsn);
> > +
> > + prev_lsn = change->lsn;
> > I did not understand, how this change is relavent to this patch
> >
>
> This is just to ensure that changes are in LSN order. I think as we
> are merging the changes before commit for streaming, it is good to
> have such an Assertion for ReorderBufferStreamTXN. And, if we want
> to have it in ReorderBufferStreamTXN, then there is no harm in keeping
> it in ReorderBufferCommit() at least to keep the code consistent. Do
> you see any problem with this?
I am fine with this.
>
> > 4.
> > + /*
> > + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> > + * information about subtransactions, which could arrive after streaming start.
> > + */
> > + if (!txn->is_schema_sent)
> > + snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
> > + txn, command_id);
> >
> > In which case, txn->is_schema_sent will be true, because at the end of
> > the stream in ReorderBufferExecuteInvalidations we are always setting
> > it false,
> > so while sending next stream it will always be false. That means we
> > never required snapshot_now variable in ReorderBufferTXN.
> >
>
> You are probably right, but as discussed we need to change this part
> of design/code (when to send schema changes) due to the issues
> discovered. So, I think this part will anyway change when we fix that
> problem.
Make sense.
>
> > 5.
> > @@ -2299,6 +2746,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer
> > *rb, TransactionId xid,
> > txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
> >
> > txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
> > +
> > + /*
> > + * We read catalog changes from WAL, which are not yet sent, so
> > + * invalidate current schema in order output plugin can resend
> > + * schema again.
> > + */
> > + txn->is_schema_sent = false;
> >
> > Same as point 4, during decode time it will never be true.
> >
>
> Sure, my previous point's reply applies here as well.
ok
>
> > 6.
> > + /* send fields */
> > + pq_sendint64(out, commit_lsn);
> > + pq_sendint64(out, txn->end_lsn);
> > + pq_sendint64(out, txn->commit_time);
> >
> > Commit_time and end_lsn is used in standby_feedback
> >
>
> I don't understand what you mean by this. Can you be a bit more clear?
I think I paste it here by mistake. just ignore it.
>
> >
> > 7.
> > + /* FIXME optimize the search by bsearch on sorted data */
> > + for (i = nsubxacts; i > 0; i--)
> > + {
> > + if (subxacts[i - 1].xid == subxid)
> > + {
> > + subidx = (i - 1);
> > + found = true;
> > + break;
> > + }
> > + }
> > We can not rollback intermediate subtransaction without rollbacking
> > latest sub-transaction, so why do we need
> > to search in the array? It will always be the the last subxact no?
> >
>
> The same thing is already mentioned in the comments above this code
> ("XXX Or perhaps we can rely on the aborts to arrive in the reverse
> order, i.e. from the inner-most subxact (when nested)? In which case
> we could simply check the last element."). I think what you are
> saying is probably right, but we can leave this as it is for now
> because this is a minor optimization which can be done later as well
> if required. However, if you see any correctness issue, then we can
> discuss.
I think more than optimization here we have the question of whether
this loop is required at all or not. Because, by optimizing we are
not adding the complexity, infact it will be simple. I think here we
need more analysis that whether we need to traverse the array or not.
So maybe for time being we can leave this as it is.
>
> > 8.
> > + /*
> > + * send feedback to upstream
> > + *
> > + * XXX Probably should send a valid LSN. But which one?
> > + */
> > + send_feedback(InvalidXLogRecPtr, false, false);
> >
> > Why feedback is sent for every change?
> >
>
> I will study this part of the patch and let you know my opinion.
Sure.
>
> Few comments on this patch series:
>
> 0001-Immediately-WAL-log-assignments:
> ------------------------------------------------------------
>
> The commit message still refers to the old design for this patch. I
> think you need to modify the commit message as per the latest patch.
>
> 0002-Issue-individual-invalidations-with-wal_level-log
> ----------------------------------------------------------------------------
> 1.
> xact_desc_invalidations(StringInfo buf,
> {
> ..
> + else if (msg->id == SHAREDINVALSNAPSHOT_ID)
> + appendStringInfo(buf, " snapshot %u", msg->sn.relId);
>
> You have removed logging for the above cache but forgot to remove its
> reference from one of the places. Also, I think you need to add a
> comment somewhere in inval.c to say why you are writing for WAL for
> some types of invalidations and not for others?
>
> 0003-Extend-the-output-plugin-API-with-stream-methods
> --------------------------------------------------------------------------------
> 1.
> + are required, while <function>stream_message_cb</function> and
> + <function>stream_message_cb</function> are optional.
>
> stream_message_cb is mentioned twice. It seems the second one is for truncate.
>
> 2.
> size of the transaction size and network bandwidth, the transfer time
> + may significantly increase the apply lag.
>
> /size of the transaction size/size of the transaction
>
> no need to mention size twice.
>
> 3.
> + Similarly to spill-to-disk behavior, streaming is triggered when the total
> + amount of changes decoded from the WAL (for all in-progress
> transactions)
> + exceeds limit defined by <varname>logical_work_mem</varname> setting.
>
> The guc name used is wrong. /Similarly to/Similar to/
>
> 4.
> stream_start_cb_wrapper()
> {
> ..
> + /* state.report_location = apply_lsn; */
> ..
> + /* FIXME ctx->write_location = apply_lsn; */
> ..
> }
>
> See, if we can fix these and similar in the callback for the stop. I
> think we don't have final_lsn till we commit/abort. Can we compute
> before calling these API's?
>
>
> 0005-Gracefully-handle-concurrent-aborts-of-uncommitte
> ----------------------------------------------------------------------------------
> 1.
> @@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> PG_CATCH();
> {
> /* TODO: Encapsulate cleanup
> from the PG_TRY and PG_CATCH blocks */
> +
> if (iterstate)
> ReorderBufferIterTXNFinish(rb, iterstate);
>
> Spurious line change.
>
> 2. The commit message of this patch refers to Prepared transactions.
> I think that needs to be changed.
>
> 0006-Implement-streaming-mode-in-ReorderBuffer
> -------------------------------------------------------------------------
> 1.
> +
> +/* iterator for streaming (only get data from memory) */
> +static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
> +
> ReorderBuffer *rb,
> +
> ReorderBufferTXN
> *txn);
> +
> +static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
> + ReorderBuffer *rb,
> +
> ReorderBufferStreamIterTXNState * state);
> +
> +static void ReorderBufferStreamIterTXNFinish(
> +
> ReorderBuffer *rb,
> +
> ReorderBufferStreamIterTXNState * state);
>
> Do we really need to introduce new APIs for iterating over changes
> from streamed transactions? Why can't we reuse the same API's as we
> use for committed xacts?
>
> 2.
> +static void
> +ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
>
> Please write some comments atop ReorderBufferStreamCommit.
>
> 3.
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> ..
> + if (txn->snapshot_now
> == NULL)
> + {
> + dlist_iter subxact_i;
> +
> + /* make sure this transaction is streamed for the first time */
> +
> Assert(!rbtxn_is_streamed(txn));
> +
> + /* at the beginning we should have invalid command ID */
> + Assert(txn->command_id ==
> InvalidCommandId);
> +
> + dlist_foreach(subxact_i, &txn->subtxns)
> + {
> + ReorderBufferTXN *subtxn;
> +
> +
> subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
> +
> + if (subtxn->base_snapshot != NULL &&
> +
> (txn->base_snapshot == NULL ||
> + txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
> + {
> +
> txn->base_snapshot = subtxn->base_snapshot;
>
> The logic here seems to be correct, but I am not sure why it is not
> considered to purge the base snapshot before assigning the subtxn's
> snapshot and similarly, we have not purged snapshot for subtxn once we
> are done with it. I think we can use
> ReorderBufferTransferSnapToParent to replace part of the logic here.
> Do you see any reason for doing things differently here?
>
> 4. In ReorderBufferStreamTXN, why do you need to use
> ReorderBufferCopySnap to assign txn->base_snapshot to snapshot_now.
>
> 5. I see a lot of code similarity in ReorderBufferStreamTXN and
> existing ReorderBufferCommit. I understand that there are some subtle
> differences due to which we need to write this new function but can't
> we encapsulate the specific parts of code in functions and then call
> from both places. I am talking about code in different cases for
> change->action.
>
> 6. + * Note: We never stream and serialize a transaction at the same time (e
> /(e/(we
>
I will look into these comments and reply separately.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-12-13 10:05 ` Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2019-12-13 10:05 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Dec 12, 2019 at 9:45 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Dec 11, 2019 at 5:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Dec 9, 2019 at 1:27 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > I have review the patch set and here are few comments/questions
> > >
> > > 1.
> > > +static void
> > > +pg_decode_stream_change(LogicalDecodingContext *ctx,
> > > + ReorderBufferTXN *txn,
> > > + Relation relation,
> > > + ReorderBufferChange *change)
> > > +{
> > > + OutputPluginPrepareWrite(ctx, true);
> > > + appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
> > > + OutputPluginWrite(ctx, true);
> > > +}
> > >
> > > Should we show the tuple in the streamed change like we do for the
> > > pg_decode_change?
> > >
> >
> > I think so. The patch shows the message in
> > pg_decode_stream_message(), so why to prohibit showing tuple here?
> >
> > > 2. pg_logical_slot_get_changes_guts
> > > It recreate the decoding slot [ctx =
> > > CreateDecodingContext(InvalidXLogRecPtr] but doesn't set the streaming
> > > to false, should we pass a parameter to
> > > pg_logical_slot_get_changes_guts saying whether we want streamed results or not
> > >
> >
> > CreateDecodingContext internally calls StartupDecodingContext which
> > sets the value of streaming based on if the plugin has provided
> > callbacks for streaming functions. Isn't that sufficient? Why do we
> > need additional parameters here?
>
> I don't think that if plugin provides streaming function then we
> should stream. Like pgoutput plugin provides streaming function but
> we only stream if streaming is on in create subscription command. So
> I feel that should be true with any plugin.
>
How about adding a new boolean parameter (streaming) in
pg_create_logical_replication_slot()?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2019-12-30 09:40 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2019-12-30 09:40 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Dec 12, 2019 at 9:44 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
Yesterday, Tomas has posted the latest version of the patch set which
contain the fix for schema send part. Meanwhile, I was working on few
review comments/bugfixes and refactoring. I have tried to merge those
changes with the latest patch set except the refactoring related to
"0006-Implement-streaming-mode-in-ReorderBuffer" patch, because Tomas
has also made some changes in the same patch. I have created a
separate patch for the same so that we can review the changes and then
we can merge them to the main patch.
> On Wed, Dec 11, 2019 at 5:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Dec 9, 2019 at 1:27 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > I have review the patch set and here are few comments/questions
> > >
> > > 1.
> > > +static void
> > > +pg_decode_stream_change(LogicalDecodingContext *ctx,
> > > + ReorderBufferTXN *txn,
> > > + Relation relation,
> > > + ReorderBufferChange *change)
> > > +{
> > > + OutputPluginPrepareWrite(ctx, true);
> > > + appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
> > > + OutputPluginWrite(ctx, true);
> > > +}
> > >
> > > Should we show the tuple in the streamed change like we do for the
> > > pg_decode_change?
> > >
> >
> > I think so. The patch shows the message in
> > pg_decode_stream_message(), so why to prohibit showing tuple here?
Yeah, we can do that. One option is that we can directly register
"pg_decode_change" function as stream_change_cb plugin and that will
show the tuple, another option is that we can write a similar function
as pg_decode_change and change the message which includes the text
"STREAM" so that the user can distinguish between tuple from committed
transaction and the in-progress transaction.
While analyzing this solution I have encountered one more issue, the
problem is that currently, during commit time in DecodeCommit we check
whether we need to skip the changes of the transaction or not by
calling SnapBuildXactNeedsSkip but since now we support streaming so
it's possible that before commit wal arrive we might have already sent
the changes to the output plugin even though we could have skipped
those changes. So my question is instead of checking at the commit
time can't we check before adding to ReorderBuffer itself or we can
truncate the changes if SnapBuildXactNeedsSkip is true whenever
logical_decoding_workmem limit is reached.
> > Few comments on this patch series:
> >
> > 0001-Immediately-WAL-log-assignments:
> > ------------------------------------------------------------
> >
> > The commit message still refers to the old design for this patch. I
> > think you need to modify the commit message as per the latest patch.
Done
> >
> > 0002-Issue-individual-invalidations-with-wal_level-log
> > ----------------------------------------------------------------------------
> > 1.
> > xact_desc_invalidations(StringInfo buf,
> > {
> > ..
> > + else if (msg->id == SHAREDINVALSNAPSHOT_ID)
> > + appendStringInfo(buf, " snapshot %u", msg->sn.relId);
> >
> > You have removed logging for the above cache but forgot to remove its
> > reference from one of the places. Also, I think you need to add a
> > comment somewhere in inval.c to say why you are writing for WAL for
> > some types of invalidations and not for others?
Done
> >
> > 0003-Extend-the-output-plugin-API-with-stream-methods
> > --------------------------------------------------------------------------------
> > 1.
> > + are required, while <function>stream_message_cb</function> and
> > + <function>stream_message_cb</function> are optional.
> >
> > stream_message_cb is mentioned twice. It seems the second one is for truncate.
Done
> >
> > 2.
> > size of the transaction size and network bandwidth, the transfer time
> > + may significantly increase the apply lag.
> >
> > /size of the transaction size/size of the transaction
> >
> > no need to mention size twice.
Done
> >
> > 3.
> > + Similarly to spill-to-disk behavior, streaming is triggered when the total
> > + amount of changes decoded from the WAL (for all in-progress
> > transactions)
> > + exceeds limit defined by <varname>logical_work_mem</varname> setting.
> >
> > The guc name used is wrong. /Similarly to/Similar to/
Done
> >
> > 4.
> > stream_start_cb_wrapper()
> > {
> > ..
> > + /* state.report_location = apply_lsn; */
> > ..
> > + /* FIXME ctx->write_location = apply_lsn; */
> > ..
> > }
> >
> > See, if we can fix these and similar in the callback for the stop. I
> > think we don't have final_lsn till we commit/abort. Can we compute
> > before calling these API's?
Done
> >
> >
> > 0005-Gracefully-handle-concurrent-aborts-of-uncommitte
> > ----------------------------------------------------------------------------------
> > 1.
> > @@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> > PG_CATCH();
> > {
> > /* TODO: Encapsulate cleanup
> > from the PG_TRY and PG_CATCH blocks */
> > +
> > if (iterstate)
> > ReorderBufferIterTXNFinish(rb, iterstate);
> >
> > Spurious line change.
> >
Done
> > 2. The commit message of this patch refers to Prepared transactions.
> > I think that needs to be changed.
> >
> > 0006-Implement-streaming-mode-in-ReorderBuffer
> > -------------------------------------------------------------------------
> > 1.
> > +
> > +/* iterator for streaming (only get data from memory) */
> > +static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
> > +
> > ReorderBuffer *rb,
> > +
> > ReorderBufferTXN
> > *txn);
> > +
> > +static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
> > + ReorderBuffer *rb,
> > +
> > ReorderBufferStreamIterTXNState * state);
> > +
> > +static void ReorderBufferStreamIterTXNFinish(
> > +
> > ReorderBuffer *rb,
> > +
> > ReorderBufferStreamIterTXNState * state);
> >
> > Do we really need to introduce new APIs for iterating over changes
> > from streamed transactions? Why can't we reuse the same API's as we
> > use for committed xacts?
Done
> >
> > 2.
> > +static void
> > +ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
> >
> > Please write some comments atop ReorderBufferStreamCommit.
Done
> >
> > 3.
> > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > {
> > ..
> > ..
> > + if (txn->snapshot_now
> > == NULL)
> > + {
> > + dlist_iter subxact_i;
> > +
> > + /* make sure this transaction is streamed for the first time */
> > +
> > Assert(!rbtxn_is_streamed(txn));
> > +
> > + /* at the beginning we should have invalid command ID */
> > + Assert(txn->command_id ==
> > InvalidCommandId);
> > +
> > + dlist_foreach(subxact_i, &txn->subtxns)
> > + {
> > + ReorderBufferTXN *subtxn;
> > +
> > +
> > subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
> > +
> > + if (subtxn->base_snapshot != NULL &&
> > +
> > (txn->base_snapshot == NULL ||
> > + txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
> > + {
> > +
> > txn->base_snapshot = subtxn->base_snapshot;
> >
> > The logic here seems to be correct, but I am not sure why it is not
> > considered to purge the base snapshot before assigning the subtxn's
> > snapshot and similarly, we have not purged snapshot for subtxn once we
> > are done with it. I think we can use
> > ReorderBufferTransferSnapToParent to replace part of the logic here.
> > Do you see any reason for doing things differently here?
Done
> >
> > 4. In ReorderBufferStreamTXN, why do you need to use
> > ReorderBufferCopySnap to assign txn->base_snapshot to snapshot_now.
IMHO, here instead of directly copying the base snapshot we are
modifying it by passing command id and thats the reason we are copying
it.
> >
> > 5. I see a lot of code similarity in ReorderBufferStreamTXN and
> > existing ReorderBufferCommit. I understand that there are some subtle
> > differences due to which we need to write this new function but can't
> > we encapsulate the specific parts of code in functions and then call
> > from both places. I am talking about code in different cases for
> > change->action.
Done
> >
> > 6. + * Note: We never stream and serialize a transaction at the same time (e
> > /(e/(we
Done
I have also found one bug in
"v3-0012-fixup-add-proper-schema-tracking.patch" due to which some of
the streaming test cases were failing, I have created a separate patch
to fix the same.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v4-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.3K) download
From 4fd022529dc2c4f5c4d96c0e69537ae5f40684e7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v4 01/19] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 5353b6a..708e523 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index aa9dca0..a8a8084 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 67418b0..4435c63 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index bc532d0..897b755 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 9d2899d..5b9740c 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 3fea199..b1976ac 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 0193611..a676151 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index 9375e54..bcfba0a 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v4-0002-Issue-individual-invalidations-with-wal_level-log.patch (application/octet-stream, 15.7K) download
From c067b209589fa6288c6aef48955a21bd8ee2732d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v4 02/19] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 50 ++++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 +++++++++
src/backend/replication/logical/reorderbuffer.c | 56 +++++++++++++++++---
src/backend/utils/cache/inval.c | 69 +++++++++++++++++++++++++
src/include/access/xact.h | 18 ++++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 229 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index 4c411c5..7ecdc0e 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,44 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 708e523..da15556 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 897b755..9bcefb6 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 53affeb..b1feff3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -464,6 +464,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1804,17 +1805,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2209,6 +2216,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2656,6 +2695,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2752,6 +2792,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3027,6 +3068,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index f09e3a9..0682c55 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -104,6 +104,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +211,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +493,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +517,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +539,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +559,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1546,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 5b9740c..82d4942 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0867ee9..6a7187b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v4-0003-fixup-is_schema_sent-set-too-early.patch (application/octet-stream, 858B) download
From 5c284042e2a21f7f7ed7bdc84a5eccec77e4c05f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 22:50:55 +0100
Subject: [PATCH v4 03/19] fixup: is_schema_sent set too early
---
src/backend/replication/logical/reorderbuffer.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index b1feff3..c0b9725 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1819,7 +1819,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
--
1.8.3.1
v4-0004-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From 8a7ab4d4c83e2453ea926727c3c7985d4e0f7222 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v4 04/19] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 6c33c4b..9c77791 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..ace21ec 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 7e06615..57edf54 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 6879a2e..1e934d2 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index d4ce54f..a305462 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6a7187b..5b4be2b 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v4-0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structur.patch (application/octet-stream, 8.1K) download
From 5ebe0327a929d10d4fa82f27e4828dc60c6e2701 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH v4 05/19] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++---------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c0b9725..f74c199 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -732,7 +732,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -752,7 +752,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -775,7 +775,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -835,7 +835,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -851,7 +851,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1061,7 +1061,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1090,7 +1090,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1256,7 +1256,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1304,7 +1304,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1319,7 +1319,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1336,7 +1336,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1969,7 +1969,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2117,7 +2117,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2296,7 +2296,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2313,7 +2313,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2333,7 +2333,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2521,12 +2521,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5b4be2b..19c7bac 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
v4-0006-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (application/octet-stream, 11.9K) download
From 8b0ad280cae050bae5e6229cb3485967db919bb3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v4 06/19] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 34 ++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++++--
src/include/utils/snapmgr.h | 4 ++-
6 files changed, 109 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index ace21ec..319349a 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0128bb3..a27eac0 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "improper heap_getnext call");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1431,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "improper heap_fetch call");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1552,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "improper heap_hot_search_buffer call");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1708,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "improper heap_get_latest_tid call");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5514,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "improper heap_hot_search call");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index 2599b5d..201acfb 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,17 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +525,17 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +662,17 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f74c199..3e87597 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -683,7 +683,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1533,7 +1533,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1784,7 +1784,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1804,7 +1804,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 47b0517..9fa1e43 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index 67b07df..9a8f9ce 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v4-0007-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 44.3K) download
From 4bc6427ef3439096a34d12dabb28ba60d09045d7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:42:31 +0530
Subject: [PATCH v4 07/19] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 1075 ++++++++++++++++++++++-
src/include/replication/reorderbuffer.h | 32 +
3 files changed, 1112 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index 3e36467..cf10dd0 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3e87597..232d9f4 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,6 +149,28 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
+/*
+ * k-way in-order change iteration support structures
+ *
+ * This is a simplified version for streaming, which does not require
+ * serialization to files and only reads changes that are currently in
+ * memory.
+ */
+typedef struct ReorderBufferStreamIterTXNEntry
+{
+ XLogRecPtr lsn;
+ ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+} ReorderBufferStreamIterTXNEntry;
+
+typedef struct ReorderBufferStreamIterTXNState
+{
+ binaryheap *heap;
+ Size nr_txns;
+ dlist_head old_change;
+ ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
+} ReorderBufferStreamIterTXNState;
+
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -213,6 +235,20 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+/* iterator for streaming (only get data from memory) */
+static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
+static void ReorderBufferStreamIterTXNFinish(
+ ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -227,6 +263,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -235,6 +272,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -362,6 +408,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -759,6 +808,33 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+static void
+AssertChangeLsnOrder(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -855,6 +931,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -978,7 +1057,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1006,6 +1085,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter cur_txn_i;
int32 off;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1020,6 +1102,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1235,6 +1320,210 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
+ * Binary heap comparison function (streaming iterator).
+ */
+static int
+ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
+{
+ ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
+ XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
+ XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
+
+ if (pos_a < pos_b)
+ return 1;
+ else if (pos_a == pos_b)
+ return 0;
+ return -1;
+}
+
+/*
+ * Allocate & initialize an iterator which iterates in lsn order over a
+ * transaction and all its subtransactions. This version is meant for
+ * streaming of incomplete transactions.
+ */
+static ReorderBufferStreamIterTXNState *
+ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Size nr_txns = 0;
+ ReorderBufferStreamIterTXNState *state;
+ dlist_iter cur_txn_i;
+ int32 off;
+
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(rb, txn);
+
+ /*
+ * Calculate the size of our heap: one element for every transaction that
+ * contains changes. (Besides the transactions already in the reorder
+ * buffer, we count the one we were directly passed.)
+ */
+ if (txn->nentries > 0)
+ nr_txns++;
+
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(rb, cur_txn);
+
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
+
+ /*
+ * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
+ * need to allocate/build a heap then.
+ */
+
+ /* allocate iteration state */
+ state = (ReorderBufferStreamIterTXNState *)
+ MemoryContextAllocZero(rb->context,
+ sizeof(ReorderBufferStreamIterTXNState) +
+ sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
+
+ state->nr_txns = nr_txns;
+ dlist_init(&state->old_change);
+
+ /* allocate heap */
+ state->heap = binaryheap_allocate(state->nr_txns,
+ ReorderBufferStreamIterCompare,
+ state);
+
+ /*
+ * Now insert items into the binary heap, in an unordered fashion. (We
+ * will run a heap assembly step at the end; this is more efficient.)
+ */
+
+ off = 0;
+
+ /* add toplevel transaction if it contains changes */
+ if (txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+
+ if (cur_txn->nentries > 0)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ &cur_txn->changes);
+
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
+
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
+ }
+
+ Assert(off == nr_txns);
+
+ /* assemble a valid binary heap */
+ binaryheap_build(state->heap);
+
+ return state;
+}
+
+/*
+ * Return the next change when iterating over a transaction and its
+ * subtransactions.
+ *
+ * Returns NULL when no further changes exist.
+ */
+static ReorderBufferChange *
+ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
+{
+ ReorderBufferChange *change;
+ ReorderBufferStreamIterTXNEntry *entry;
+ int32 off;
+
+ /* nothing there anymore */
+ if (state->heap->bh_size == 0)
+ return NULL;
+
+ off = DatumGetInt32(binaryheap_first(state->heap));
+ entry = &state->entries[off];
+
+ /* free memory we might have "leaked" in the previous *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ change = entry->change;
+
+ /*
+ * update heap with information about which transaction has the next
+ * relevant change in LSN order
+ */
+
+ /* there are in-memory changes */
+ if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ {
+ dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ ReorderBufferChange *next_change =
+ dlist_container(ReorderBufferChange, node, next);
+
+ /* txn stays the same */
+ state->entries[off].lsn = next_change->lsn;
+ state->entries[off].change = next_change;
+
+ binaryheap_replace_first(state->heap, Int32GetDatum(off));
+ return change;
+ }
+
+ /* ok, no changes there anymore, remove */
+ binaryheap_remove_first(state->heap);
+
+ return change;
+}
+
+/*
+ * Deallocate the iterator
+ */
+static void
+ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
+ ReorderBufferStreamIterTXNState * state)
+{
+ /* free memory we might have "leaked" in the last *Next call */
+ if (!dlist_is_empty(&state->old_change))
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node,
+ dlist_pop_head_node(&state->old_change));
+ ReorderBufferReturnChange(rb, change);
+ Assert(dlist_is_empty(&state->old_change));
+ }
+
+ binaryheap_free(state->heap);
+ pfree(state);
+}
+
+/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1327,33 +1616,104 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
*/
static void
-ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- dlist_iter iter;
- HASHCTL hash_ctl;
+ dlist_mutable_iter iter;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
- memset(&hash_ctl, 0, sizeof(hash_ctl));
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
- hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
- hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
- hash_ctl.hcxt = rb->context;
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
- /*
- * create the hash with the exact number of to-be-stored tuplecids from
- * the start
- */
- txn->tuplecid_hash =
- hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
- dlist_foreach(iter, &txn->tuplecids)
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
+ * Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
+ */
+static void
+ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_iter iter;
+ HASHCTL hash_ctl;
+
+ memset(&hash_ctl, 0, sizeof(hash_ctl));
+
+ hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
+ hash_ctl.entrysize = sizeof(ReorderBufferTupleCidEnt);
+ hash_ctl.hcxt = rb->context;
+
+ /*
+ * create the hash with the exact number of to-be-stored tuplecids from
+ * the start
+ */
+ txn->tuplecid_hash =
+ hash_create("ReorderBufferTupleCid", txn->ntuplecids, &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ dlist_foreach(iter, &txn->tuplecids)
{
ReorderBufferTupleCidKey key;
ReorderBufferTupleCidEnt *ent;
@@ -1403,6 +1763,16 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
}
+static void
+ReorderBufferDestroyTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+}
+
/*
* Copy a provided snapshot so we can modify it privately. This is needed so
* that catalog modifying transactions can look into intermediate catalog
@@ -1476,6 +1846,19 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
/*
* Perform the replay of a transaction and its non-aborted subtransactions.
*
@@ -1515,6 +1898,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
txn->origin_lsn = origin_lsn;
/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
* If this transaction has no snapshot, it didn't make any changes to the
* database, so there's nothing to decode. Note that
* ReorderBufferCommitChild will have transferred any snapshots from
@@ -1549,6 +1948,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
@@ -1565,6 +1965,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1928,6 +2338,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2012,6 +2429,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2147,8 +2571,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2156,6 +2589,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2167,19 +2601,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2208,6 +2651,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2283,6 +2727,9 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+
+ /* Invalidate current schema as well */
+ txn->is_schema_sent = false;
}
/*
@@ -2297,6 +2744,23 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * We read catalog changes from WAL, which are not yet sent, so
+ * invalidate current schema in order output plugin can resend
+ * schema again.
+ */
+ txn->is_schema_sent = false;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ {
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+ txn->toptxn->is_schema_sent = false;
+ }
}
/*
@@ -2401,6 +2865,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2420,15 +2916,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2721,6 +3248,498 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+ bool using_subtxn;
+ Size streamed = 0;
+ ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
+
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+ ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+
+ if (subtxn->base_snapshot != NULL &&
+ (txn->base_snapshot == NULL ||
+ txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
+ {
+ txn->base_snapshot = subtxn->base_snapshot;
+ txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
+ subtxn->base_snapshot = NULL;
+ subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
+ }
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
+
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+
+ /*
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
+ *
+ * When we're called via the SQL SRF there's already a transaction
+ * started, so start an explicit subtransaction there.
+ */
+ using_subtxn = IsTransactionOrTransactionBlock();
+
+ PG_TRY();
+ {
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *change;
+ ReorderBufferChange *specinsert = NULL;
+
+ if (using_subtxn)
+ BeginInternalSubTransaction("stream");
+ else
+ StartTransactionCommand();
+
+ /* start streaming this chunk of transaction */
+ rb->stream_start(rb, txn);
+
+ iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
+ while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
+ {
+ Relation relation = NULL;
+ Oid reloid;
+
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ if (prev_lsn != InvalidXLogRecPtr)
+ Assert(prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* we're going to stream this change */
+ streamed++;
+
+ switch (change->action)
+ {
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+
+ /*
+ * Confirmation for speculative insertion arrived. Simply
+ * use as a normal record. It'll be cleaned up at the end
+ * of INSERT processing.
+ */
+ Assert(specinsert->data.tp.oldtuple == NULL);
+ change = specinsert;
+ change->action = REORDER_BUFFER_CHANGE_INSERT;
+
+ /* intentionally fall through */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ Assert(snapshot_now);
+
+ reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
+ change->data.tp.relnode.relNode);
+
+ /*
+ * Catalog tuple without data, emitted while catalog was
+ * in the process of being rewritten.
+ */
+ if (reloid == InvalidOid &&
+ change->data.tp.newtuple == NULL &&
+ change->data.tp.oldtuple == NULL)
+ goto change_done;
+ else if (reloid == InvalidOid)
+ elog(ERROR, "could not map filenode \"%s\" to relation OID",
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ relation = RelationIdGetRelation(reloid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
+ reloid,
+ relpathperm(change->data.tp.relnode,
+ MAIN_FORKNUM));
+
+ if (!RelationIsLogicallyLogged(relation))
+ goto change_done;
+
+ /*
+ * For now ignore sequence changes entirely. Most of the
+ * time they don't log changes using records we
+ * understand, so it doesn't make sense to handle the few
+ * cases we do.
+ */
+ if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
+ goto change_done;
+
+ /* user-triggered change */
+ if (!IsToastRelation(relation))
+ {
+ ReorderBufferToastReplace(rb, txn, relation, change);
+ rb->stream_change(rb, txn, relation, change);
+
+ /*
+ * Only clear reassembled toast chunks if we're sure
+ * they're not required anymore. The creator of the
+ * tuple tells us.
+ */
+ if (change->data.tp.clear_toast_afterwards)
+ ReorderBufferToastReset(rb, txn);
+ }
+ /* we're not interested in toast deletions */
+ else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+ {
+ /*
+ * Need to reassemble the full toasted Datum in
+ * memory, to ensure the chunks don't get reused till
+ * we're done remove it from the list of this
+ * transaction's changes. Otherwise it will get
+ * freed/reused while restoring spooled data from
+ * disk.
+ */
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
+ }
+
+ change_done:
+
+ /*
+ * Either speculative insertion was confirmed, or it was
+ * unsuccessful and the record isn't needed anymore.
+ */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ if (relation != NULL)
+ {
+ RelationClose(relation);
+ relation = NULL;
+ }
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+
+ /*
+ * Speculative insertions are dealt with by delaying the
+ * processing of the insert until the confirmation record
+ * arrives. For that we simply unlink the record from the
+ * chain, so it does not get freed/reused while restoring
+ * spooled data from disk.
+ *
+ * This is safe in the face of concurrent catalog changes
+ * because the relevant relation can't be changed between
+ * speculative insertion and confirmation due to
+ * CheckTableNotInUse() and locking.
+ */
+
+ /* clear out a pending (and thus failed) speculation */
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* and memorize the pending insertion */
+ dlist_delete(&change->node);
+ specinsert = change;
+ break;
+
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ {
+ int i;
+ int nrelids = change->data.truncate.nrelids;
+ int nrelations = 0;
+ Relation *relations;
+
+ relations = palloc0(nrelids * sizeof(Relation));
+ for (i = 0; i < nrelids; i++)
+ {
+ Oid relid = change->data.truncate.relids[i];
+ Relation relation;
+
+ relation = RelationIdGetRelation(relid);
+
+ if (relation == NULL)
+ elog(ERROR, "could not open relation with OID %u", relid);
+
+ if (!RelationIsLogicallyLogged(relation))
+ continue;
+
+ relations[nrelations++] = relation;
+ }
+
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ for (i = 0; i < nrelations; i++)
+ RelationClose(relations[i]);
+
+ break;
+ }
+
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ /* get rid of the old */
+ TeardownHistoricSnapshot(false);
+
+ if (snapshot_now->copied)
+ {
+ ReorderBufferFreeSnap(rb, snapshot_now);
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+
+ /*
+ * Restored from disk, need to be careful not to double
+ * free. We could introduce refcounting for that, but for
+ * now this seems infrequent enough not to care.
+ */
+ else if (change->data.snapshot->copied)
+ {
+ snapshot_now =
+ ReorderBufferCopySnap(rb, change->data.snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ snapshot_now = change->data.snapshot;
+ }
+
+ /*
+ * TOCHECK: Snapshot changed, then invalidate current schema to reflect
+ * possible catalog changes.
+ */
+ txn->is_schema_sent = false;
+
+ /* and continue with the new one */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ Assert(change->data.command_id != InvalidCommandId);
+
+ if (command_id < change->data.command_id)
+ {
+ command_id = change->data.command_id;
+
+ if (!snapshot_now->copied)
+ {
+ /* we don't use the global one anymore */
+ snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+
+ snapshot_now->curcid = command_id;
+
+ TeardownHistoricSnapshot(false);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
+ }
+
+ break;
+
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ elog(ERROR, "tuplecid value in changequeue");
+ break;
+ }
+ }
+
+ /*
+ * There's a speculative insertion remaining, just clean in up, it
+ * can't have been successful, otherwise we'd gotten a confirmation
+ * record.
+ */
+ if (specinsert)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+
+ /* clean up the iterator */
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+ iterstate = NULL;
+
+ /* call stream_stop callback */
+ rb->stream_stop(rb, txn);
+
+ /* this is just a sanity check against bad output plugin behaviour */
+ if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
+ elog(ERROR, "output plugin used XID %u",
+ GetCurrentTransactionId());
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+
+ /* cleanup */
+ TeardownHistoricSnapshot(false);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
+
+ /*
+ * Aborting the current (sub-)transaction as a whole has the right
+ * semantics. We want all locks acquired in here to be released, not
+ * reassigned to the parent and we do not want any database access
+ * have persistent effects.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+ }
+ PG_CATCH();
+ {
+ /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
+ if (iterstate)
+ ReorderBufferStreamIterTXNFinish(rb, iterstate);
+
+ TeardownHistoricSnapshot(true);
+
+ /*
+ * Force cache invalidation to happen outside of a valid transaction
+ * to prevent catalog access as we just caught an error.
+ */
+ AbortCurrentTransaction();
+
+ /* make sure there's no cache pollution */
+ ReorderBufferExecuteInvalidations(rb, txn);
+
+ if (using_subtxn)
+ RollbackAndReleaseCurrentSubTransaction();
+
+ PG_RE_THROW();
+ }
+ PG_END_TRY();
+
+ /*
+ * Discard the changes that we just streamed, and mark the transactions
+ * as streamed (if they contained changes).
+ */
+ ReorderBufferTruncateTXN(rb, txn);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 19c7bac..7d08e2f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Do we need to send schema for this transaction in output plugin?
+ */
+ bool is_schema_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v4-0008-fixup-add-is_schema_sent-back.patch (application/octet-stream, 854B) download
From dcb320769da1aa3c4bd7c80c3dcee22a0fb3ed33 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:02:38 +0100
Subject: [PATCH v4 08/19] fixup: add is_schema_sent back
---
src/backend/replication/logical/reorderbuffer.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 232d9f4..78b5c00 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2229,6 +2229,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
--
1.8.3.1
v4-0009-fixup-get-rid-of-is_schema_sent-entirely.patch (application/octet-stream, 3.6K) download
From 1f6b7447ac3d80fc47a603d106b2d26bfc46c710 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:46:15 +0100
Subject: [PATCH v4 09/19] fixup: get rid of is_schema_sent entirely
We'll do this in the pgoutput.c code directly, not in reorderbuffer.
---
src/backend/replication/logical/reorderbuffer.c | 26 ++-----------------------
src/include/replication/reorderbuffer.h | 5 -----
2 files changed, 2 insertions(+), 29 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 78b5c00..dda651e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2229,7 +2229,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
@@ -2728,9 +2727,6 @@ ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
for (i = 0; i < txn->ninvalidations; i++)
LocalExecuteInvalidationMessage(&txn->invalidations[i]);
-
- /* Invalidate current schema as well */
- txn->is_schema_sent = false;
}
/*
@@ -2747,21 +2743,11 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
/*
- * We read catalog changes from WAL, which are not yet sent, so
- * invalidate current schema in order output plugin can resend
- * schema again.
- */
- txn->is_schema_sent = false;
-
- /*
* TOCHECK: Mark toplevel transaction as having catalog changes too
* if one of its children has.
*/
if (txn->toptxn != NULL)
- {
txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
- txn->toptxn->is_schema_sent = false;
- }
}
/*
@@ -3344,9 +3330,8 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* TOCHECK: We have to rebuild historic snapshot to be sure it includes all
* information about subtransactions, which could arrive after streaming start.
*/
- if (!txn->is_schema_sent)
- snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
- txn, command_id);
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
}
/*
@@ -3601,12 +3586,6 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
snapshot_now = change->data.snapshot;
}
- /*
- * TOCHECK: Snapshot changed, then invalidate current schema to reflect
- * possible catalog changes.
- */
- txn->is_schema_sent = false;
-
/* and continue with the new one */
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
txn->xid);
@@ -3645,7 +3624,6 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 7d08e2f..e2b8db0 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -237,11 +237,6 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
- * Do we need to send schema for this transaction in output plugin?
- */
- bool is_schema_sent;
-
- /*
* Toplevel transaction for this subxact (NULL for top-level).
*/
struct ReorderBufferTXN *toptxn;
--
1.8.3.1
v4-0010-Support-logical_decoding_work_mem-set-from-create.patch (application/octet-stream, 13.1K) download
From 218d6f72deea3db0109db556086d73d0092cf9e7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v4 10/19] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5d1c902..8b1923c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 68d88ff..2a27648 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 5408edc..fbb4473 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 545d2fc..0ab6855 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 63ba0ae..c80acd3 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1729,6 +1729,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 3483c1b..cf6e03b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3cb13d8..10ea113 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 41714ea..1db706a 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
v4-0011-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 89.6K) download
From 7e4f8b8f47289299f5b856f0e77925d4b8cbb166 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:05:20 +0100
Subject: [PATCH v4 11/19] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 263 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2027 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 2a27648..15a6f5a 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index fbb4473..b2b93d6 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 7410b2f..a479ce9 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 0ab6855..9970170 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index c57b578..0a013ed 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 57edf54..30a6ee4 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47d..5a379fb 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index c80acd3..cf053e9 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -529,6 +636,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -694,6 +1122,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -814,6 +1245,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -913,6 +1347,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1004,6 +1441,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1101,6 +1554,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1116,6 +1585,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1564,6 +2036,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1730,6 +2760,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index cf6e03b..8490ea4 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,16 +45,42 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
bool replicate_valid;
PublicationActions pubactions;
@@ -64,6 +90,7 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
@@ -84,16 +111,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +199,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +228,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +252,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +273,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +361,42 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = txn->is_schema_sent;
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +422,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ txn->is_schema_sent = true;
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +450,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +482,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +499,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +508,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +534,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +558,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +639,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -623,6 +834,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+ }
+
+}
+
+/*
* Relcache invalidation callback
*/
static void
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ba08ad4..8eb3160 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 1f23665..2e0743a 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -969,6 +969,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 10ea113..8793676 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index f2e873d..c522703 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 3fc430a..bf02cbc 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 1db706a..3d19b5d 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v4-0012-fixup-add-proper-schema-tracking.patch (application/octet-stream, 3.3K) download
From 13f056ac3452588e6f8e3e832f222f3f6416857f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:56:04 +0100
Subject: [PATCH v4 12/19] fixup: add proper schema tracking
---
src/backend/replication/pgoutput/pgoutput.c | 45 +++++++++++++++++++++++++++--
1 file changed, 43 insertions(+), 2 deletions(-)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 8490ea4..0148f4c 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -82,6 +82,8 @@ typedef struct RelationSyncEntry
Oid relid; /* relation oid */
TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -96,6 +98,11 @@ static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -366,6 +373,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
{
bool schema_sent;
TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
/*
* Remember XID of the (sub)transaction for the change. We don't care if
@@ -378,6 +386,11 @@ maybe_send_schema(LogicalDecodingContext *ctx,
if (in_streaming)
xid = change->txn->xid;
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
/*
* Do we need to send the schema? We do track streamed transactions
* separately, because those may not be applied later (and the regular
@@ -391,7 +404,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
* occur when streaming already started, so we have to track new catalog
* changes somehow.
*/
- schema_sent = txn->is_schema_sent;
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
}
else
schema_sent = relentry->schema_sent;
@@ -432,7 +445,7 @@ maybe_send_schema(LogicalDecodingContext *ctx,
relentry->xid = change->txn->xid;
if (in_streaming)
- txn->is_schema_sent = true;
+ set_schema_sent_in_streamed_txn(relentry, topxid);
else
relentry->schema_sent = true;
}
@@ -760,6 +773,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*/
static RelationSyncEntry *
--
1.8.3.1
v4-0013-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From 88f482bd96c5ffbdc30e1dc76c79ddabd4ecd554 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 2 Dec 2019 09:58:50 +0530
Subject: [PATCH v4 13/19] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dcb5811..180ea88 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1996,6 +1996,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f7800f0..5897611 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -779,7 +779,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index dda651e..1949dc4 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -358,6 +358,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3709,6 +3713,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
PG_END_TRY();
/*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
+ /*
* Discard the changes that we just streamed, and mark the transactions
* as streamed (if they contained changes).
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 2e0743a..9f93f11 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1293,7 +1293,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1314,7 +1314,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2357,6 +2358,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3185,7 +3189,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3242,6 +3246,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3265,6 +3272,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3351,6 +3361,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3598,12 +3613,19 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillTxns = rb->spillTxns;
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
+
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index ac8f64b..3b897a5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5166,9 +5166,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e2b8db0..e132c3c 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -506,15 +506,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index a6b3205..7efc332 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 80a0782..5ab21a8 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1955,9 +1955,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v4-0014-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From a3d35c1aeb5b8dc4d2f810958966f5bc689bfd06 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v4 14/19] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 77a1560..8cd1993 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -65,7 +65,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 81547f6..8dfeafc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v4-0015-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1013B) download
From 441194d7823933ca81bf8cfb86d4780e91e166eb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v4 15/19] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 1949dc4..a3e2c06 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1544,6 +1544,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
v4-0016-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 403e8a2caee4f6ec7be98ac3c0ac8b60fb5af952 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v4 16/19] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v4-0017-Extend-handling-of-concurrent-aborts-for-streamin.patch (application/octet-stream, 3.7K) download
From 4030d805e9358a374f522c2c11c0ac15e36e19a7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 22 Nov 2019 12:43:38 +0530
Subject: [PATCH v4 17/19] Extend handling of concurrent aborts for streaming
transaction
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 5 ++++
2 files changed, 38 insertions(+), 3 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index a3e2c06..789b425 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2348,9 +2348,9 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
/*
* When the (sub)transaction was streamed, notify the remote node
- * about the abort.
+ * about the abort only if we have sent any data for this transaction.
*/
- if (rbtxn_is_streamed(txn))
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
rb->stream_abort(rb, txn, lsn);
/* cosmetic... */
@@ -3266,6 +3266,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
volatile CommandId command_id;
bool using_subtxn;
Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
@@ -3395,6 +3396,13 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* we're going to stream this change */
streamed++;
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -3456,6 +3464,10 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferToastReplace(rb, txn, relation, change);
rb->stream_change(rb, txn, relation, change);
+ /* Remember that we have sent some data for this txn.*/
+ if (!change->txn->any_data_sent)
+ change->txn->any_data_sent = true;
+
/*
* Only clear reassembled toast chunks if we're sure
* they're not required anymore. The creator of the
@@ -3694,6 +3706,9 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferStreamIterTXNFinish(rb, iterstate);
@@ -3712,7 +3727,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
}
PG_END_TRY();
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e132c3c..6186465 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -237,6 +237,11 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
* Toplevel transaction for this subxact (NULL for top-level).
*/
struct ReorderBufferTXN *toptxn;
--
1.8.3.1
v4-0018-Review-comment-fix-and-refactoring.patch (application/octet-stream, 36.1K) download
From 76cea7d3a261cde0492547db75e4c95ea787ad32 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Sun, 29 Dec 2019 15:41:26 +0530
Subject: [PATCH v4 18/19] Review comment fix and refactoring
---
src/backend/replication/logical/reorderbuffer.c | 995 ++++++------------------
1 file changed, 237 insertions(+), 758 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 789b425..8b3f112 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -149,28 +149,6 @@ typedef struct ReorderBufferIterTXNState
ReorderBufferIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
} ReorderBufferIterTXNState;
-/*
- * k-way in-order change iteration support structures
- *
- * This is a simplified version for streaming, which does not require
- * serialization to files and only reads changes that are currently in
- * memory.
- */
-typedef struct ReorderBufferStreamIterTXNEntry
-{
- XLogRecPtr lsn;
- ReorderBufferChange *change;
- ReorderBufferTXN *txn;
-} ReorderBufferStreamIterTXNEntry;
-
-typedef struct ReorderBufferStreamIterTXNState
-{
- binaryheap *heap;
- Size nr_txns;
- dlist_head old_change;
- ReorderBufferStreamIterTXNEntry entries[FLEXIBLE_ARRAY_MEMBER];
-} ReorderBufferStreamIterTXNState;
-
/* toast datastructures */
typedef struct ReorderBufferToastEnt
{
@@ -235,20 +213,6 @@ static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
-
-/* iterator for streaming (only get data from memory) */
-static ReorderBufferStreamIterTXNState * ReorderBufferStreamIterTXNInit(
- ReorderBuffer *rb,
- ReorderBufferTXN *txn);
-
-static ReorderBufferChange *ReorderBufferStreamIterTXNNext(
- ReorderBuffer *rb,
- ReorderBufferStreamIterTXNState * state);
-
-static void ReorderBufferStreamIterTXNFinish(
- ReorderBuffer *rb,
- ReorderBufferStreamIterTXNState * state);
-
/*
* ---------------------------------------
* Disk serialization support functions
@@ -1324,210 +1288,6 @@ ReorderBufferIterTXNFinish(ReorderBuffer *rb,
}
/*
- * Binary heap comparison function (streaming iterator).
- */
-static int
-ReorderBufferStreamIterCompare(Datum a, Datum b, void *arg)
-{
- ReorderBufferStreamIterTXNState *state = (ReorderBufferStreamIterTXNState *) arg;
- XLogRecPtr pos_a = state->entries[DatumGetInt32(a)].lsn;
- XLogRecPtr pos_b = state->entries[DatumGetInt32(b)].lsn;
-
- if (pos_a < pos_b)
- return 1;
- else if (pos_a == pos_b)
- return 0;
- return -1;
-}
-
-/*
- * Allocate & initialize an iterator which iterates in lsn order over a
- * transaction and all its subtransactions. This version is meant for
- * streaming of incomplete transactions.
- */
-static ReorderBufferStreamIterTXNState *
-ReorderBufferStreamIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn)
-{
- Size nr_txns = 0;
- ReorderBufferStreamIterTXNState *state;
- dlist_iter cur_txn_i;
- int32 off;
-
- /* Check ordering of changes in the toplevel transaction. */
- AssertChangeLsnOrder(rb, txn);
-
- /*
- * Calculate the size of our heap: one element for every transaction that
- * contains changes. (Besides the transactions already in the reorder
- * buffer, we count the one we were directly passed.)
- */
- if (txn->nentries > 0)
- nr_txns++;
-
- dlist_foreach(cur_txn_i, &txn->subtxns)
- {
- ReorderBufferTXN *cur_txn;
-
- cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
-
- /* Check ordering of changes in this subtransaction. */
- AssertChangeLsnOrder(rb, cur_txn);
-
- if (cur_txn->nentries > 0)
- nr_txns++;
- }
-
- /*
- * TODO: Consider adding fastpath for the rather common nr_txns=1 case, no
- * need to allocate/build a heap then.
- */
-
- /* allocate iteration state */
- state = (ReorderBufferStreamIterTXNState *)
- MemoryContextAllocZero(rb->context,
- sizeof(ReorderBufferStreamIterTXNState) +
- sizeof(ReorderBufferStreamIterTXNEntry) * nr_txns);
-
- state->nr_txns = nr_txns;
- dlist_init(&state->old_change);
-
- /* allocate heap */
- state->heap = binaryheap_allocate(state->nr_txns,
- ReorderBufferStreamIterCompare,
- state);
-
- /*
- * Now insert items into the binary heap, in an unordered fashion. (We
- * will run a heap assembly step at the end; this is more efficient.)
- */
-
- off = 0;
-
- /* add toplevel transaction if it contains changes */
- if (txn->nentries > 0)
- {
- ReorderBufferChange *cur_change;
-
- cur_change = dlist_head_element(ReorderBufferChange, node,
- &txn->changes);
-
- state->entries[off].lsn = cur_change->lsn;
- state->entries[off].change = cur_change;
- state->entries[off].txn = txn;
-
- binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
- }
-
- /* add subtransactions if they contain changes */
- dlist_foreach(cur_txn_i, &txn->subtxns)
- {
- ReorderBufferTXN *cur_txn;
-
- cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
-
- if (cur_txn->nentries > 0)
- {
- ReorderBufferChange *cur_change;
-
- cur_change = dlist_head_element(ReorderBufferChange, node,
- &cur_txn->changes);
-
- state->entries[off].lsn = cur_change->lsn;
- state->entries[off].change = cur_change;
- state->entries[off].txn = cur_txn;
-
- binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
- }
- }
-
- Assert(off == nr_txns);
-
- /* assemble a valid binary heap */
- binaryheap_build(state->heap);
-
- return state;
-}
-
-/*
- * Return the next change when iterating over a transaction and its
- * subtransactions.
- *
- * Returns NULL when no further changes exist.
- */
-static ReorderBufferChange *
-ReorderBufferStreamIterTXNNext(ReorderBuffer *rb, ReorderBufferStreamIterTXNState * state)
-{
- ReorderBufferChange *change;
- ReorderBufferStreamIterTXNEntry *entry;
- int32 off;
-
- /* nothing there anymore */
- if (state->heap->bh_size == 0)
- return NULL;
-
- off = DatumGetInt32(binaryheap_first(state->heap));
- entry = &state->entries[off];
-
- /* free memory we might have "leaked" in the previous *Next call */
- if (!dlist_is_empty(&state->old_change))
- {
- change = dlist_container(ReorderBufferChange, node,
- dlist_pop_head_node(&state->old_change));
- ReorderBufferReturnChange(rb, change);
- Assert(dlist_is_empty(&state->old_change));
- }
-
- change = entry->change;
-
- /*
- * update heap with information about which transaction has the next
- * relevant change in LSN order
- */
-
- /* there are in-memory changes */
- if (dlist_has_next(&entry->txn->changes, &entry->change->node))
- {
- dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
- ReorderBufferChange *next_change =
- dlist_container(ReorderBufferChange, node, next);
-
- /* txn stays the same */
- state->entries[off].lsn = next_change->lsn;
- state->entries[off].change = next_change;
-
- binaryheap_replace_first(state->heap, Int32GetDatum(off));
- return change;
- }
-
- /* ok, no changes there anymore, remove */
- binaryheap_remove_first(state->heap);
-
- return change;
-}
-
-/*
- * Deallocate the iterator
- */
-static void
-ReorderBufferStreamIterTXNFinish(ReorderBuffer *rb,
- ReorderBufferStreamIterTXNState * state)
-{
- /* free memory we might have "leaked" in the last *Next call */
- if (!dlist_is_empty(&state->old_change))
- {
- ReorderBufferChange *change;
-
- change = dlist_container(ReorderBufferChange, node,
- dlist_pop_head_node(&state->old_change));
- ReorderBufferReturnChange(rb, change);
- Assert(dlist_is_empty(&state->old_change));
- }
-
- binaryheap_free(state->heap);
- pfree(state);
-}
-
-/*
* Cleanup the contents of a transaction, usually after the transaction
* committed or aborted.
*/
@@ -1854,6 +1614,11 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
SnapBuildSnapDecRefcount(snap);
}
+/*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ */
static void
ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
@@ -1868,86 +1633,38 @@ ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
*
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
bool using_subtxn;
+ Size streamed = 0;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
-
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
+ ReorderBufferBuildTupleCidHash(rb, txn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ /* setup the initial snapshot */
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
- * If the transaction was (partially) streamed, we need to commit it in a
- * 'streamed' way. That is, we first stream the remaining part of the
- * transaction, and then invoke stream_commit message.
- *
- * XXX Called after everything (origin ID and LSN, ...) is stored in the
- * transaction, so we don't pass that directly.
- *
- * XXX Somewhat hackish redirection, perhaps needs to be refactored?
- */
- if (rbtxn_is_streamed(txn))
- {
- ReorderBufferStreamCommit(rb, txn);
- return;
- }
-
- /*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
- */
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
-
- snapshot_now = txn->base_snapshot;
-
- /* build data to be able to lookup the CommandIds of catalog tuples */
- ReorderBufferBuildTupleCidHash(rb, txn);
-
- /* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
-
- /*
- * Decoding needs access to syscaches et al., which in turn use
- * heavyweight locks and such. Thus we need to have enough state around to
- * keep track of those. The easiest way is to simply use a transaction
- * internally. That also allows us to easily enforce that nothing writes
- * to the database by checking for xid assignments.
+ * Decoding needs access to syscaches et al., which in turn use
+ * heavyweight locks and such. Thus we need to have enough state around to
+ * keep track of those. The easiest way is to simply use a transaction
+ * internally. That also allows us to easily enforce that nothing writes
+ * to the database by checking for xid assignments.
*
* When we're called via the SQL SRF there's already a transaction
* started, so start an explicit subtransaction there.
@@ -1961,11 +1678,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
iterstate = ReorderBufferIterTXNInit(rb, txn);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1978,11 +1699,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* subtransactions. The changes may have the same LSN due to
* MULTI_INSERT xlog records.
*/
- if (prev_lsn != InvalidXLogRecPtr)
- Assert(prev_lsn <= change->lsn);
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
prev_lsn = change->lsn;
+ if (streaming)
+ {
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
+ /* Increment the stream count. */
+ streamed++;
+ }
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1992,8 +1725,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -2059,7 +1790,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -2080,8 +1819,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -2099,7 +1836,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -2157,7 +1894,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -2166,10 +1911,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -2200,9 +1951,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -2222,7 +1973,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -2260,16 +2012,46 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
- /* cleanup */
- TeardownHistoricSnapshot(false);
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
/*
* Aborting the current (sub-)transaction as a whole has the right
@@ -2285,14 +2067,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then Discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -2311,18 +2101,117 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -3264,10 +3153,6 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
volatile Snapshot snapshot_now;
volatile CommandId command_id;
- bool using_subtxn;
- Size streamed = 0;
- MemoryContext ccxt = CurrentMemoryContext;
- ReorderBufferStreamIterTXNState *volatile iterstate = NULL;
/*
* If this is a subxact, we need to stream the top-level transaction
@@ -3305,16 +3190,7 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferTXN *subtxn;
subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
-
- if (subtxn->base_snapshot != NULL &&
- (txn->base_snapshot == NULL ||
- txn->base_snapshot_lsn > subtxn->base_snapshot_lsn))
- {
- txn->base_snapshot = subtxn->base_snapshot;
- txn->base_snapshot_lsn = subtxn->base_snapshot_lsn;
- subtxn->base_snapshot = NULL;
- subtxn->base_snapshot_lsn = InvalidXLogRecPtr;
- }
+ ReorderBufferTransferSnapToParent(txn, subtxn);
}
command_id = FirstCommandId;
@@ -3344,407 +3220,10 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
- * build data to be able to lookup the CommandIds of catalog tuples
- */
- ReorderBufferBuildTupleCidHash(rb, txn);
-
- /* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
-
- /*
- * Decoding needs access to syscaches et al., which in turn use
- * heavyweight locks and such. Thus we need to have enough state around to
- * keep track of those. The easiest way is to simply use a transaction
- * internally. That also allows us to easily enforce that nothing writes
- * to the database by checking for xid assignments.
- *
- * When we're called via the SQL SRF there's already a transaction
- * started, so start an explicit subtransaction there.
+ * Access the main routine to decode the changes and send to output plugin.
*/
- using_subtxn = IsTransactionOrTransactionBlock();
-
- PG_TRY();
- {
- XLogRecPtr prev_lsn = InvalidXLogRecPtr;
- ReorderBufferChange *change;
- ReorderBufferChange *specinsert = NULL;
-
- if (using_subtxn)
- BeginInternalSubTransaction("stream");
- else
- StartTransactionCommand();
-
- /* start streaming this chunk of transaction */
- rb->stream_start(rb, txn);
-
- iterstate = ReorderBufferStreamIterTXNInit(rb, txn);
- while ((change = ReorderBufferStreamIterTXNNext(rb, iterstate)) != NULL)
- {
- Relation relation = NULL;
- Oid reloid;
-
- /*
- * Enforce correct ordering of changes, merged from multiple
- * subtransactions. The changes may have the same LSN due to
- * MULTI_INSERT xlog records.
- */
- if (prev_lsn != InvalidXLogRecPtr)
- Assert(prev_lsn <= change->lsn);
-
- prev_lsn = change->lsn;
-
- /* we're going to stream this change */
- streamed++;
-
- /*
- * Set the CheckXidAlive to the current (sub)xid for which this
- * change belongs to so that we can detect the abort while we are
- * decoding.
- */
- CheckXidAlive = change->txn->xid;
-
- switch (change->action)
- {
- case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
-
- /*
- * Confirmation for speculative insertion arrived. Simply
- * use as a normal record. It'll be cleaned up at the end
- * of INSERT processing.
- */
- Assert(specinsert->data.tp.oldtuple == NULL);
- change = specinsert;
- change->action = REORDER_BUFFER_CHANGE_INSERT;
-
- /* intentionally fall through */
- case REORDER_BUFFER_CHANGE_INSERT:
- case REORDER_BUFFER_CHANGE_UPDATE:
- case REORDER_BUFFER_CHANGE_DELETE:
- Assert(snapshot_now);
-
- reloid = RelidByRelfilenode(change->data.tp.relnode.spcNode,
- change->data.tp.relnode.relNode);
-
- /*
- * Catalog tuple without data, emitted while catalog was
- * in the process of being rewritten.
- */
- if (reloid == InvalidOid &&
- change->data.tp.newtuple == NULL &&
- change->data.tp.oldtuple == NULL)
- goto change_done;
- else if (reloid == InvalidOid)
- elog(ERROR, "could not map filenode \"%s\" to relation OID",
- relpathperm(change->data.tp.relnode,
- MAIN_FORKNUM));
-
- relation = RelationIdGetRelation(reloid);
-
- if (relation == NULL)
- elog(ERROR, "could not open relation with OID %u (for filenode \"%s\")",
- reloid,
- relpathperm(change->data.tp.relnode,
- MAIN_FORKNUM));
-
- if (!RelationIsLogicallyLogged(relation))
- goto change_done;
-
- /*
- * For now ignore sequence changes entirely. Most of the
- * time they don't log changes using records we
- * understand, so it doesn't make sense to handle the few
- * cases we do.
- */
- if (relation->rd_rel->relkind == RELKIND_SEQUENCE)
- goto change_done;
-
- /* user-triggered change */
- if (!IsToastRelation(relation))
- {
- ReorderBufferToastReplace(rb, txn, relation, change);
- rb->stream_change(rb, txn, relation, change);
-
- /* Remember that we have sent some data for this txn.*/
- if (!change->txn->any_data_sent)
- change->txn->any_data_sent = true;
-
- /*
- * Only clear reassembled toast chunks if we're sure
- * they're not required anymore. The creator of the
- * tuple tells us.
- */
- if (change->data.tp.clear_toast_afterwards)
- ReorderBufferToastReset(rb, txn);
- }
- /* we're not interested in toast deletions */
- else if (change->action == REORDER_BUFFER_CHANGE_INSERT)
- {
- /*
- * Need to reassemble the full toasted Datum in
- * memory, to ensure the chunks don't get reused till
- * we're done remove it from the list of this
- * transaction's changes. Otherwise it will get
- * freed/reused while restoring spooled data from
- * disk.
- */
- dlist_delete(&change->node);
- ReorderBufferToastAppendChunk(rb, txn, relation,
- change);
- }
-
- change_done:
-
- /*
- * Either speculative insertion was confirmed, or it was
- * unsuccessful and the record isn't needed anymore.
- */
- if (specinsert != NULL)
- {
- ReorderBufferReturnChange(rb, specinsert);
- specinsert = NULL;
- }
-
- if (relation != NULL)
- {
- RelationClose(relation);
- relation = NULL;
- }
- break;
-
- case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
-
- /*
- * Speculative insertions are dealt with by delaying the
- * processing of the insert until the confirmation record
- * arrives. For that we simply unlink the record from the
- * chain, so it does not get freed/reused while restoring
- * spooled data from disk.
- *
- * This is safe in the face of concurrent catalog changes
- * because the relevant relation can't be changed between
- * speculative insertion and confirmation due to
- * CheckTableNotInUse() and locking.
- */
-
- /* clear out a pending (and thus failed) speculation */
- if (specinsert != NULL)
- {
- ReorderBufferReturnChange(rb, specinsert);
- specinsert = NULL;
- }
-
- /* and memorize the pending insertion */
- dlist_delete(&change->node);
- specinsert = change;
- break;
-
- case REORDER_BUFFER_CHANGE_TRUNCATE:
- {
- int i;
- int nrelids = change->data.truncate.nrelids;
- int nrelations = 0;
- Relation *relations;
-
- relations = palloc0(nrelids * sizeof(Relation));
- for (i = 0; i < nrelids; i++)
- {
- Oid relid = change->data.truncate.relids[i];
- Relation relation;
-
- relation = RelationIdGetRelation(relid);
-
- if (relation == NULL)
- elog(ERROR, "could not open relation with OID %u", relid);
-
- if (!RelationIsLogicallyLogged(relation))
- continue;
-
- relations[nrelations++] = relation;
- }
-
- rb->stream_truncate(rb, txn, nrelations, relations, change);
-
- for (i = 0; i < nrelations; i++)
- RelationClose(relations[i]);
-
- break;
- }
-
- case REORDER_BUFFER_CHANGE_MESSAGE:
-
- rb->stream_message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
- break;
-
- case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
- /* get rid of the old */
- TeardownHistoricSnapshot(false);
-
- if (snapshot_now->copied)
- {
- ReorderBufferFreeSnap(rb, snapshot_now);
- snapshot_now =
- ReorderBufferCopySnap(rb, change->data.snapshot,
- txn, command_id);
- }
-
- /*
- * Restored from disk, need to be careful not to double
- * free. We could introduce refcounting for that, but for
- * now this seems infrequent enough not to care.
- */
- else if (change->data.snapshot->copied)
- {
- snapshot_now =
- ReorderBufferCopySnap(rb, change->data.snapshot,
- txn, command_id);
- }
- else
- {
- snapshot_now = change->data.snapshot;
- }
-
- /* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
- txn->xid);
- break;
-
- case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
- Assert(change->data.command_id != InvalidCommandId);
-
- if (command_id < change->data.command_id)
- {
- command_id = change->data.command_id;
-
- if (!snapshot_now->copied)
- {
- /* we don't use the global one anymore */
- snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
- txn, command_id);
- }
-
- snapshot_now->curcid = command_id;
-
- TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
- txn->xid);
- }
-
- break;
-
- case REORDER_BUFFER_CHANGE_INVALIDATION:
-
- /*
- * Execute the invalidation message locally.
- *
- * XXX Do we need to care about relcacheInitFileInval and
- * the other fields added to ReorderBufferChange, or just
- * about the message itself?
- */
- LocalExecuteInvalidationMessage(&change->data.inval.msg);
- break;
-
- case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
- elog(ERROR, "tuplecid value in changequeue");
- break;
- }
- }
-
- /*
- * There's a speculative insertion remaining, just clean in up, it
- * can't have been successful, otherwise we'd gotten a confirmation
- * record.
- */
- if (specinsert)
- {
- ReorderBufferReturnChange(rb, specinsert);
- specinsert = NULL;
- }
-
- /* clean up the iterator */
- ReorderBufferStreamIterTXNFinish(rb, iterstate);
- iterstate = NULL;
-
- /* call stream_stop callback */
- rb->stream_stop(rb, txn);
-
- /* this is just a sanity check against bad output plugin behaviour */
- if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
- elog(ERROR, "output plugin used XID %u",
- GetCurrentTransactionId());
-
- /* remember the command ID and snapshot for the streaming run */
- txn->command_id = command_id;
- txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
- txn, command_id);
-
- /* cleanup */
- TeardownHistoricSnapshot(false);
-
- /*
- * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
- * any memory. We could also keep the hash table and update it with
- * new ctid values, but this seems simpler and good enough for now.
- */
- ReorderBufferDestroyTupleCidHash(rb, txn);
-
- /*
- * Aborting the current (sub-)transaction as a whole has the right
- * semantics. We want all locks acquired in here to be released, not
- * reassigned to the parent and we do not want any database access
- * have persistent effects.
- */
- AbortCurrentTransaction();
-
- /* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
-
- if (using_subtxn)
- RollbackAndReleaseCurrentSubTransaction();
- }
- PG_CATCH();
- {
- MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
- ErrorData *errdata = CopyErrorData();
-
- /* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
- if (iterstate)
- ReorderBufferStreamIterTXNFinish(rb, iterstate);
-
- TeardownHistoricSnapshot(true);
-
- /*
- * Force cache invalidation to happen outside of a valid transaction
- * to prevent catalog access as we just caught an error.
- */
- AbortCurrentTransaction();
-
- /* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
-
- if (using_subtxn)
- RollbackAndReleaseCurrentSubTransaction();
-
- /* re-throw only if it's not an abort */
- if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
- {
- MemoryContextSwitchTo(ecxt);
- PG_RE_THROW();
- }
- else
- {
- /* remember the command ID and snapshot for the streaming run */
- txn->command_id = command_id;
- txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
- txn, command_id);
- rb->stream_stop(rb, txn);
-
- FlushErrorState();
- }
- }
- PG_END_TRY();
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
/*
* Update the stream statistics.
--
1.8.3.1
v4-0019-Bugfix-in-schema-tracking.patch (application/octet-stream, 837B) download
From 036f61804046977873257075143af41b7abe2875 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 30 Dec 2019 13:14:25 +0530
Subject: [PATCH v4 19/19] Bugfix in schema-tracking
---
src/backend/replication/pgoutput/pgoutput.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 0148f4c..ffd9f94 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -937,7 +937,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-04 10:37 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 04:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-22 05:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 3 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-04 10:37 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Dec 30, 2019 at 3:11 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Dec 12, 2019 at 9:44 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> Yesterday, Tomas has posted the latest version of the patch set which
> contain the fix for schema send part. Meanwhile, I was working on few
> review comments/bugfixes and refactoring. I have tried to merge those
> changes with the latest patch set except the refactoring related to
> "0006-Implement-streaming-mode-in-ReorderBuffer" patch, because Tomas
> has also made some changes in the same patch.
>
I don't see any changes by Tomas in that particular patch, am I
missing something?
> I have created a
> separate patch for the same so that we can review the changes and then
> we can merge them to the main patch.
>
It is better to merge it with the main patch for
"Implement-streaming-mode-in-ReorderBuffer", otherwise, it is a bit
difficult to review.
> > > 0002-Issue-individual-invalidations-with-wal_level-log
> > > ----------------------------------------------------------------------------
> > > 1.
> > > xact_desc_invalidations(StringInfo buf,
> > > {
> > > ..
> > > + else if (msg->id == SHAREDINVALSNAPSHOT_ID)
> > > + appendStringInfo(buf, " snapshot %u", msg->sn.relId);
> > >
> > > You have removed logging for the above cache but forgot to remove its
> > > reference from one of the places. Also, I think you need to add a
> > > comment somewhere in inval.c to say why you are writing for WAL for
> > > some types of invalidations and not for others?
> Done
>
I don't see any new comments as asked by me. I think we should also
consider WAL logging at each command end instead of doing piecemeal as
discussed in another email [1], which will have lesser code changes
and maybe better in performance. You might want to evaluate the
performance of both approaches.
> > >
> > > 0003-Extend-the-output-plugin-API-with-stream-methods
> > > --------------------------------------------------------------------------------
> > >
> > > 4.
> > > stream_start_cb_wrapper()
> > > {
> > > ..
> > > + /* state.report_location = apply_lsn; */
> > > ..
> > > + /* FIXME ctx->write_location = apply_lsn; */
> > > ..
> > > }
> > >
> > > See, if we can fix these and similar in the callback for the stop. I
> > > think we don't have final_lsn till we commit/abort. Can we compute
> > > before calling these API's?
> Done
>
You have just used final_lsn, but I don't see where you have ensured
that it is set before the API stream_stop_cb_wrapper. I think we need
something similar to what Vignesh has done in one of his bug-fix patch
[2]. See my comment below in this regard.
> > >
> > >
> > > 0005-Gracefully-handle-concurrent-aborts-of-uncommitte
> > > ----------------------------------------------------------------------------------
> > > 1.
> > > @@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> > > PG_CATCH();
> > > {
> > > /* TODO: Encapsulate cleanup
> > > from the PG_TRY and PG_CATCH blocks */
> > > +
> > > if (iterstate)
> > > ReorderBufferIterTXNFinish(rb, iterstate);
> > >
> > > Spurious line change.
> > >
> Done
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "improper heap_getnext call");
Earlier, I thought we don't need to check if it is a regular table in
this check, but it is required because output plugins can try to do
that and if they do so during decoding (with historic snapshots), the
same should be not allowed.
How about changing the error message to "unexpected heap_getnext call
during logical decoding" or something like that?
> > > 2. The commit message of this patch refers to Prepared transactions.
> > > I think that needs to be changed.
> > >
> > > 0006-Implement-streaming-mode-in-ReorderBuffer
> > > -------------------------------------------------------------------------
Few comments on v4-0018-Review-comment-fix-and-refactoring:
1.
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
Shouldn't we try to final_lsn as is done by Vignesh's patch [2]?
2.
+ if (streaming)
+ {
+ /*
+ * Set the CheckXidAlive to the current (sub)xid for which this
+ * change belongs to so that we can detect the abort while we are
+ * decoding.
+ */
+ CheckXidAlive = change->txn->xid;
+
+ /* Increment the stream count. */
+ streamed++;
+ }
Is the variable 'streamed' used anywhere?
3.
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ ReorderBufferDestroyTupleCidHash(rb, txn);
Won't this be required only when we are streaming changes?
As per my understanding apart from the above comments, the known
pending work for this patchset is as follows:
a. The two open items agreed to you in the email [3].
b. Complete the handling of schema_sent as discussed above [4].
c. Few comments by Vignesh and the response on the same by me [5][6].
d. WAL overhead and performance testing for additional WAL logging by
this patchset.
e. Some way to see the tuple for streamed transactions by decoding API
as speculated by you [7].
Have I missed anything?
[1] - https://www.postgresql.org/message-id/CAA4eK1LOa%2B2KqNX%3Dm%3D1qMBDW%2Bo50AuwjAOX6ZqL-rWGiH1F9MQ%40...
[2] - https://www.postgresql.org/message-id/CALDaNm3MDxFnsZsnSqVhPBLS3%3DqzNH6%2BYzB%3DxYuX2vbtsUeFgw%40ma...
[3] - https://www.postgresql.org/message-id/CAFiTN-uT5YZE0egGhKdTteTjcGrPi8hb%3DFMPpr9_hEB7hozQ-Q%40mail.g...
[4] - https://www.postgresql.org/message-id/CAA4eK1KjD9x0mS4JxzCbu3gu-r6K7XJRV%2BZcGb3BH6U3x2uxew%40mail.g...
[5] - https://www.postgresql.org/message-id/CALDaNm0DNUojjt7CV-fa59_kFbQQ3rcMBtauvo44ttea7r9KaA%40mail.gma...
[6] - https://www.postgresql.org/message-id/CAA4eK1%2BZvupW00c--dqEg8f3dHZDOGmA9xOQLyQHjRSoDi6AkQ%40mail.g...
[7] - https://www.postgresql.org/message-id/CAFiTN-t8PmKA1X4jEqKmkvs0ggWpy0APWpPuaJwpx2YpfAf97w%40mail.gma...
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-06 03:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-06 03:50 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Jan 4, 2020 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Dec 30, 2019 at 3:11 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Dec 12, 2019 at 9:44 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > Yesterday, Tomas has posted the latest version of the patch set which
> > contain the fix for schema send part. Meanwhile, I was working on few
> > review comments/bugfixes and refactoring. I have tried to merge those
> > changes with the latest patch set except the refactoring related to
> > "0006-Implement-streaming-mode-in-ReorderBuffer" patch, because Tomas
> > has also made some changes in the same patch.
> >
>
> I don't see any changes by Tomas in that particular patch, am I
> missing something?
He has created some sub-patch from the main patch for handling
schema-sent issue. So if I make change in that patch all other
patches will conflict.
>
> > I have created a
> > separate patch for the same so that we can review the changes and then
> > we can merge them to the main patch.
> >
>
> It is better to merge it with the main patch for
> "Implement-streaming-mode-in-ReorderBuffer", otherwise, it is a bit
> difficult to review.
Actually, we can merge 0008, 0009, 0012, 0018 to the main patch
(0007). Basically, if we merge all of them then we don't need to deal
with the conflict. I think Tomas has kept them separate so that we
can review the solution for the schema sent. And, I kept 0018 as a
separate patch to avoid conflict and rebasing in 0008, 0009 and 0012.
In the next patch set, I will merge all of them to 0007.
>
> > > > 0002-Issue-individual-invalidations-with-wal_level-log
> > > > ----------------------------------------------------------------------------
> > > > 1.
> > > > xact_desc_invalidations(StringInfo buf,
> > > > {
> > > > ..
> > > > + else if (msg->id == SHAREDINVALSNAPSHOT_ID)
> > > > + appendStringInfo(buf, " snapshot %u", msg->sn.relId);
> > > >
> > > > You have removed logging for the above cache but forgot to remove its
> > > > reference from one of the places. Also, I think you need to add a
> > > > comment somewhere in inval.c to say why you are writing for WAL for
> > > > some types of invalidations and not for others?
> > Done
> >
>
> I don't see any new comments as asked by me.
Oh, I just fixed one part of the comment and overlooked the rest. Will fix.
I think we should also
> consider WAL logging at each command end instead of doing piecemeal as
> discussed in another email [1], which will have lesser code changes
> and maybe better in performance. You might want to evaluate the
> performance of both approaches.
Ok
>
> > > >
> > > > 0003-Extend-the-output-plugin-API-with-stream-methods
> > > > --------------------------------------------------------------------------------
> > > >
> > > > 4.
> > > > stream_start_cb_wrapper()
> > > > {
> > > > ..
> > > > + /* state.report_location = apply_lsn; */
> > > > ..
> > > > + /* FIXME ctx->write_location = apply_lsn; */
> > > > ..
> > > > }
> > > >
> > > > See, if we can fix these and similar in the callback for the stop. I
> > > > think we don't have final_lsn till we commit/abort. Can we compute
> > > > before calling these API's?
> > Done
> >
>
> You have just used final_lsn, but I don't see where you have ensured
> that it is set before the API stream_stop_cb_wrapper. I think we need
> something similar to what Vignesh has done in one of his bug-fix patch
> [2]. See my comment below in this regard.
You can refer below hunk in 0018.
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
>
> > > >
> > > >
> > > > 0005-Gracefully-handle-concurrent-aborts-of-uncommitte
> > > > ----------------------------------------------------------------------------------
> > > > 1.
> > > > @@ -1877,6 +1877,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> > > > PG_CATCH();
> > > > {
> > > > /* TODO: Encapsulate cleanup
> > > > from the PG_TRY and PG_CATCH blocks */
> > > > +
> > > > if (iterstate)
> > > > ReorderBufferIterTXNFinish(rb, iterstate);
> > > >
> > > > Spurious line change.
> > > >
> > Done
>
> + /*
> + * We don't expect direct calls to heap_getnext with valid
> + * CheckXidAlive for regular tables. Track that below.
> + */
> + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> + !(IsCatalogRelation(scan->rs_base.rs_rd) ||
> + RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
> + elog(ERROR, "improper heap_getnext call");
>
> Earlier, I thought we don't need to check if it is a regular table in
> this check, but it is required because output plugins can try to do
> that
I did not understand that, can you give some example?
and if they do so during decoding (with historic snapshots), the
> same should be not allowed.
>
> How about changing the error message to "unexpected heap_getnext call
> during logical decoding" or something like that?
Ok
>
> > > > 2. The commit message of this patch refers to Prepared transactions.
> > > > I think that needs to be changed.
> > > >
> > > > 0006-Implement-streaming-mode-in-ReorderBuffer
> > > > -------------------------------------------------------------------------
>
> Few comments on v4-0018-Review-comment-fix-and-refactoring:
> 1.
> + if (streaming)
> + {
> + /*
> + * Set the last last of the stream as the final lsn before calling
> + * stream stop.
> + */
> + txn->final_lsn = prev_lsn;
> + rb->stream_stop(rb, txn);
> + }
>
> Shouldn't we try to final_lsn as is done by Vignesh's patch [2]?
Isn't it the same, there we are doing while serializing and here we
are doing while streaming? Basically, the last LSN we streamed. Am I
missing something?
>
> 2.
> + if (streaming)
> + {
> + /*
> + * Set the CheckXidAlive to the current (sub)xid for which this
> + * change belongs to so that we can detect the abort while we are
> + * decoding.
> + */
> + CheckXidAlive = change->txn->xid;
> +
> + /* Increment the stream count. */
> + streamed++;
> + }
>
> Is the variable 'streamed' used anywhere?
>
> 3.
> + /*
> + * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
> + * any memory. We could also keep the hash table and update it with
> + * new ctid values, but this seems simpler and good enough for now.
> + */
> + ReorderBufferDestroyTupleCidHash(rb, txn);
>
> Won't this be required only when we are streaming changes?
I will work on this review comments and reply to them separately along
with the patch.
>
> As per my understanding apart from the above comments, the known
> pending work for this patchset is as follows:
> a. The two open items agreed to you in the email [3].
> b. Complete the handling of schema_sent as discussed above [4].
> c. Few comments by Vignesh and the response on the same by me [5][6].
> d. WAL overhead and performance testing for additional WAL logging by
> this patchset.
> e. Some way to see the tuple for streamed transactions by decoding API
> as speculated by you [7].
>
> Have I missed anything?
I think this is the list I remember. Apart from these few points by
Robert which are still under discussion[8].
>
> [1] - https://www.postgresql.org/message-id/CAA4eK1LOa%2B2KqNX%3Dm%3D1qMBDW%2Bo50AuwjAOX6ZqL-rWGiH1F9MQ%40...
> [2] - https://www.postgresql.org/message-id/CALDaNm3MDxFnsZsnSqVhPBLS3%3DqzNH6%2BYzB%3DxYuX2vbtsUeFgw%40ma...
> [3] - https://www.postgresql.org/message-id/CAFiTN-uT5YZE0egGhKdTteTjcGrPi8hb%3DFMPpr9_hEB7hozQ-Q%40mail.g...
> [4] - https://www.postgresql.org/message-id/CAA4eK1KjD9x0mS4JxzCbu3gu-r6K7XJRV%2BZcGb3BH6U3x2uxew%40mail.g...
> [5] - https://www.postgresql.org/message-id/CALDaNm0DNUojjt7CV-fa59_kFbQQ3rcMBtauvo44ttea7r9KaA%40mail.gma...
> [6] - https://www.postgresql.org/message-id/CAA4eK1%2BZvupW00c--dqEg8f3dHZDOGmA9xOQLyQHjRSoDi6AkQ%40mail.g...
> [7] - https://www.postgresql.org/message-id/CAFiTN-t8PmKA1X4jEqKmkvs0ggWpy0APWpPuaJwpx2YpfAf97w%40mail.gma...
[8] https://www.postgresql.org/message-id/CA%2BTgmoYH6N_YDvKH9AaAJo5ZTHn142K%3DB75VO9yKvjjjHcoZhA%40mail...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-06 08:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-06 08:41 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 9:21 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sat, Jan 4, 2020 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > It is better to merge it with the main patch for
> > "Implement-streaming-mode-in-ReorderBuffer", otherwise, it is a bit
> > difficult to review.
> Actually, we can merge 0008, 0009, 0012, 0018 to the main patch
> (0007). Basically, if we merge all of them then we don't need to deal
> with the conflict. I think Tomas has kept them separate so that we
> can review the solution for the schema sent. And, I kept 0018 as a
> separate patch to avoid conflict and rebasing in 0008, 0009 and 0012.
> In the next patch set, I will merge all of them to 0007.
>
Okay, I think we can merge those patches.
> >
> > + /*
> > + * We don't expect direct calls to heap_getnext with valid
> > + * CheckXidAlive for regular tables. Track that below.
> > + */
> > + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> > + !(IsCatalogRelation(scan->rs_base.rs_rd) ||
> > + RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
> > + elog(ERROR, "improper heap_getnext call");
> >
> > Earlier, I thought we don't need to check if it is a regular table in
> > this check, but it is required because output plugins can try to do
> > that
> I did not understand that, can you give some example?
>
I think it can lead to the same problem of concurrent aborts as for
catalog scans.
> >
> > > > > 2. The commit message of this patch refers to Prepared transactions.
> > > > > I think that needs to be changed.
> > > > >
> > > > > 0006-Implement-streaming-mode-in-ReorderBuffer
> > > > > -------------------------------------------------------------------------
> >
> > Few comments on v4-0018-Review-comment-fix-and-refactoring:
> > 1.
> > + if (streaming)
> > + {
> > + /*
> > + * Set the last last of the stream as the final lsn before calling
> > + * stream stop.
> > + */
> > + txn->final_lsn = prev_lsn;
> > + rb->stream_stop(rb, txn);
> > + }
> >
> > Shouldn't we try to final_lsn as is done by Vignesh's patch [2]?
> Isn't it the same, there we are doing while serializing and here we
> are doing while streaming? Basically, the last LSN we streamed. Am I
> missing something?
>
No, I think you are right.
Few more comments:
--------------------------------
v4-0007-Implement-streaming-mode-in-ReorderBuffer
1.
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
..
+ /*
+ * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
+ * information about
subtransactions, which could arrive after streaming start.
+ */
+ if (!txn->is_schema_sent)
+ snapshot_now
= ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn,
command_id);
..
}
Why are we using base snapshot here instead of the snapshot we saved
the first time streaming has happened? And as mentioned in comments,
won't we need to consider the snapshots for subtransactions that
arrived after the last time we have streamed the changes?
2.
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn-
>snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+
txn, command_id);
I don't see where the txn->snapshot_now is getting freed. The
base_snapshot is freed in ReorderBufferCleanupTXN, but I don't see
this getting freed.
3.
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
..
+ /*
+ * If this is a subxact, we need to stream the top-level transaction
+ * instead.
+ */
+ if (txn->toptxn)
+ {
+
ReorderBufferStreamTXN(rb, txn->toptxn);
+ return;
+ }
Is it ever possible that we reach here for subtransaction, if not,
then it should be Assert rather than if condition?
4. In ReorderBufferStreamTXN(), don't we need to set some of the txn
fields like origin_id, origin_lsn as we do in ReorderBufferCommit()
especially to cover the case when it gets called due to memory
overflow (aka via ReorderBufferCheckMemoryLimit).
v4-0017-Extend-handling-of-concurrent-aborts-for-streamin
1.
@@ -3712,7 +3727,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb,
ReorderBufferTXN *txn)
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- PG_RE_THROW();
+ /* re-throw only if it's not an abort */
+ if (errdata-
>sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+
}
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn-
>command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+
txn, command_id);
+ rb->stream_stop(rb, txn);
+
+
FlushErrorState();
+ }
Can you update comments either in the above code block or some other
place to explain what is the concurrent abort problem and how we dealt
with it? Also, please explain how the above error handling is
sufficient to address all the various scenarios (sub-transaction got
aborted when we have already sent some changes, or when we have not
sent any changes yet).
v4-0006-Gracefully-handle-concurrent-aborts-of-uncommitte
1.
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
Why here we can't use TransactionIdDidAbort? If we can't use it, then
can you add comments stating the reason of the same.
2.
/*
+ * An xid value pointing to a possibly ongoing or a prepared transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
In comments, there is a mention of a prepared transaction. Do we
allow prepared transactions to be decoded as part of this patch?
3.
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out
+ */
+ if (TransactionIdIsValid
(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
This comment just says what code below is doing, can you explain the
rationale behind this check. It would be better if it is clear by
reading comments, why we are doing this check after fetching the
tuple. I think this can refer to the comment I suggested to add for
changes in patch
v4-0017-Extend-handling-of-concurrent-aborts-for-streamin.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-06 10:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-06 10:26 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 9:21 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Sat, Jan 4, 2020 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > It is better to merge it with the main patch for
> > > "Implement-streaming-mode-in-ReorderBuffer", otherwise, it is a bit
> > > difficult to review.
> > Actually, we can merge 0008, 0009, 0012, 0018 to the main patch
> > (0007). Basically, if we merge all of them then we don't need to deal
> > with the conflict. I think Tomas has kept them separate so that we
> > can review the solution for the schema sent. And, I kept 0018 as a
> > separate patch to avoid conflict and rebasing in 0008, 0009 and 0012.
> > In the next patch set, I will merge all of them to 0007.
> >
>
> Okay, I think we can merge those patches.
ok
>
> > >
> > > + /*
> > > + * We don't expect direct calls to heap_getnext with valid
> > > + * CheckXidAlive for regular tables. Track that below.
> > > + */
> > > + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> > > + !(IsCatalogRelation(scan->rs_base.rs_rd) ||
> > > + RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
> > > + elog(ERROR, "improper heap_getnext call");
> > >
> > > Earlier, I thought we don't need to check if it is a regular table in
> > > this check, but it is required because output plugins can try to do
> > > that
> > I did not understand that, can you give some example?
> >
>
> I think it can lead to the same problem of concurrent aborts as for
> catalog scans.
Yeah, got it.
>
> > >
> > > > > > 2. The commit message of this patch refers to Prepared transactions.
> > > > > > I think that needs to be changed.
> > > > > >
> > > > > > 0006-Implement-streaming-mode-in-ReorderBuffer
> > > > > > -------------------------------------------------------------------------
> > >
> > > Few comments on v4-0018-Review-comment-fix-and-refactoring:
> > > 1.
> > > + if (streaming)
> > > + {
> > > + /*
> > > + * Set the last last of the stream as the final lsn before calling
> > > + * stream stop.
> > > + */
> > > + txn->final_lsn = prev_lsn;
> > > + rb->stream_stop(rb, txn);
> > > + }
> > >
> > > Shouldn't we try to final_lsn as is done by Vignesh's patch [2]?
> > Isn't it the same, there we are doing while serializing and here we
> > are doing while streaming? Basically, the last LSN we streamed. Am I
> > missing something?
> >
>
> No, I think you are right.
>
> Few more comments:
> --------------------------------
> v4-0007-Implement-streaming-mode-in-ReorderBuffer
> 1.
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> + /*
> + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> + * information about
> subtransactions, which could arrive after streaming start.
> + */
> + if (!txn->is_schema_sent)
> + snapshot_now
> = ReorderBufferCopySnap(rb, txn->base_snapshot,
> + txn,
> command_id);
> ..
> }
>
> Why are we using base snapshot here instead of the snapshot we saved
> the first time streaming has happened? And as mentioned in comments,
> won't we need to consider the snapshots for subtransactions that
> arrived after the last time we have streamed the changes?
>
> 2.
> + /* remember the command ID and snapshot for the streaming run */
> + txn->command_id = command_id;
> + txn-
> >snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
> +
> txn, command_id);
>
> I don't see where the txn->snapshot_now is getting freed. The
> base_snapshot is freed in ReorderBufferCleanupTXN, but I don't see
> this getting freed.
Ok, I will check that and fix.
>
> 3.
> +static void
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> + /*
> + * If this is a subxact, we need to stream the top-level transaction
> + * instead.
> + */
> + if (txn->toptxn)
> + {
> +
> ReorderBufferStreamTXN(rb, txn->toptxn);
> + return;
> + }
>
> Is it ever possible that we reach here for subtransaction, if not,
> then it should be Assert rather than if condition?
ReorderBufferCheckMemoryLimit, can call it either for the
subtransaction or for the main transaction, depends upon in which
ReorderBufferTXN you are adding the current change.
I will analyze your other comments and fix them in the next version.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-06 11:06 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-06 11:06 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 3:56 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > 3.
> > +static void
> > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > {
> > ..
> > + /*
> > + * If this is a subxact, we need to stream the top-level transaction
> > + * instead.
> > + */
> > + if (txn->toptxn)
> > + {
> > +
> > ReorderBufferStreamTXN(rb, txn->toptxn);
> > + return;
> > + }
> >
> > Is it ever possible that we reach here for subtransaction, if not,
> > then it should be Assert rather than if condition?
>
> ReorderBufferCheckMemoryLimit, can call it either for the
> subtransaction or for the main transaction, depends upon in which
> ReorderBufferTXN you are adding the current change.
>
That function has code like below:
ReorderBufferCheckMemoryLimit()
{
..
if (ReorderBufferCanStream(rb))
{
/*
* Pick the largest toplevel transaction and evict it from memory by
* streaming the already decoded part.
*/
txn = ReorderBufferLargestTopTXN(rb);
/* we know there has to be one, because the size is not zero */
Assert(txn && !txn->toptxn);
..
ReorderBufferStreamTXN(rb, txn);
..
}
How can it ReorderBufferTXN pass for subtransaction?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-06 11:14 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-06 11:14 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 4:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 3:56 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > 3.
> > > +static void
> > > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > > {
> > > ..
> > > + /*
> > > + * If this is a subxact, we need to stream the top-level transaction
> > > + * instead.
> > > + */
> > > + if (txn->toptxn)
> > > + {
> > > +
> > > ReorderBufferStreamTXN(rb, txn->toptxn);
> > > + return;
> > > + }
> > >
> > > Is it ever possible that we reach here for subtransaction, if not,
> > > then it should be Assert rather than if condition?
> >
> > ReorderBufferCheckMemoryLimit, can call it either for the
> > subtransaction or for the main transaction, depends upon in which
> > ReorderBufferTXN you are adding the current change.
> >
>
> That function has code like below:
>
> ReorderBufferCheckMemoryLimit()
> {
> ..
> if (ReorderBufferCanStream(rb))
> {
> /*
> * Pick the largest toplevel transaction and evict it from memory by
> * streaming the already decoded part.
> */
> txn = ReorderBufferLargestTopTXN(rb);
> /* we know there has to be one, because the size is not zero */
> Assert(txn && !txn->toptxn);
> ..
> ReorderBufferStreamTXN(rb, txn);
> ..
> }
>
> How can it ReorderBufferTXN pass for subtransaction?
>
Hmm, I missed it. You are right, will fix it.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-08 07:42 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-08 07:42 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 4:44 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 4:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Jan 6, 2020 at 3:56 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > 3.
> > > > +static void
> > > > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > > > {
> > > > ..
> > > > + /*
> > > > + * If this is a subxact, we need to stream the top-level transaction
> > > > + * instead.
> > > > + */
> > > > + if (txn->toptxn)
> > > > + {
> > > > +
> > > > ReorderBufferStreamTXN(rb, txn->toptxn);
> > > > + return;
> > > > + }
> > > >
> > > > Is it ever possible that we reach here for subtransaction, if not,
> > > > then it should be Assert rather than if condition?
> > >
> > > ReorderBufferCheckMemoryLimit, can call it either for the
> > > subtransaction or for the main transaction, depends upon in which
> > > ReorderBufferTXN you are adding the current change.
> > >
> >
> > That function has code like below:
> >
> > ReorderBufferCheckMemoryLimit()
> > {
> > ..
> > if (ReorderBufferCanStream(rb))
> > {
> > /*
> > * Pick the largest toplevel transaction and evict it from memory by
> > * streaming the already decoded part.
> > */
> > txn = ReorderBufferLargestTopTXN(rb);
> > /* we know there has to be one, because the size is not zero */
> > Assert(txn && !txn->toptxn);
> > ..
> > ReorderBufferStreamTXN(rb, txn);
> > ..
> > }
> >
> > How can it ReorderBufferTXN pass for subtransaction?
> >
> Hmm, I missed it. You are right, will fix it.
>
I have observed one more design issue. The problem is that when we
get a toasted chunks we remember the changes in the memory(hash table)
but don't stream until we get the actual change on the main table.
Now, the problem is that we might get the change of the toasted table
and the main table in different streams. So basically, in a stream,
if we have only got the toasted tuples then even after
ReorderBufferStreamTXN the memory usage will not be reduced.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-09 04:05 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-09 04:05 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have observed one more design issue.
>
Good observation.
> The problem is that when we
> get a toasted chunks we remember the changes in the memory(hash table)
> but don't stream until we get the actual change on the main table.
> Now, the problem is that we might get the change of the toasted table
> and the main table in different streams. So basically, in a stream,
> if we have only got the toasted tuples then even after
> ReorderBufferStreamTXN the memory usage will not be reduced.
>
I think we can't split such changes in a different stream (unless we
design an entirely new solution to send partial changes of toast
data), so we need to send them together. We can keep a flag like
data_complete in ReorderBufferTxn and mark it complete only when we
are able to assemble the entire tuple. Now, whenever, we try to
stream the changes once we reach the memory threshold, we can check
whether the data_complete flag is true, if so, then only send the
changes, otherwise, we can pick the next largest transaction. I think
we can retry it for few times and if we get the incomplete data for
multiple transactions, then we can decide to spill the transaction or
maybe we can directly spill the first largest transaction which has
incomplete data.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-09 05:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-09 05:00 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have observed one more design issue.
> >
>
> Good observation.
>
> > The problem is that when we
> > get a toasted chunks we remember the changes in the memory(hash table)
> > but don't stream until we get the actual change on the main table.
> > Now, the problem is that we might get the change of the toasted table
> > and the main table in different streams. So basically, in a stream,
> > if we have only got the toasted tuples then even after
> > ReorderBufferStreamTXN the memory usage will not be reduced.
> >
>
> I think we can't split such changes in a different stream (unless we
> design an entirely new solution to send partial changes of toast
> data), so we need to send them together. We can keep a flag like
> data_complete in ReorderBufferTxn and mark it complete only when we
> are able to assemble the entire tuple. Now, whenever, we try to
> stream the changes once we reach the memory threshold, we can check
> whether the data_complete flag is true, if so, then only send the
> changes, otherwise, we can pick the next largest transaction. I think
> we can retry it for few times and if we get the incomplete data for
> multiple transactions, then we can decide to spill the transaction or
> maybe we can directly spill the first largest transaction which has
> incomplete data.
>
Yeah, we might do something on this line. Basically, we need to mark
the top-transaction as data-incomplete if any of its subtransaction is
having data-incomplete (it will always be the latest sub-transaction
of the top transaction). Also, for streaming, we are checking the
largest top transaction whereas for spilling we just need the larget
(sub) transaction. So we also need to decide while picking the
largest top transaction for streaming, if we get a few transactions
with in-complete data then how we will go for the spill. Do we spill
all the sub-transactions under this top transaction or we will again
find the larget (sub) transaction for spilling.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-09 06:39 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 07:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-09 06:39 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 9, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > I have observed one more design issue.
> > >
> >
> > Good observation.
> >
> > > The problem is that when we
> > > get a toasted chunks we remember the changes in the memory(hash table)
> > > but don't stream until we get the actual change on the main table.
> > > Now, the problem is that we might get the change of the toasted table
> > > and the main table in different streams. So basically, in a stream,
> > > if we have only got the toasted tuples then even after
> > > ReorderBufferStreamTXN the memory usage will not be reduced.
> > >
> >
> > I think we can't split such changes in a different stream (unless we
> > design an entirely new solution to send partial changes of toast
> > data), so we need to send them together. We can keep a flag like
> > data_complete in ReorderBufferTxn and mark it complete only when we
> > are able to assemble the entire tuple. Now, whenever, we try to
> > stream the changes once we reach the memory threshold, we can check
> > whether the data_complete flag is true, if so, then only send the
> > changes, otherwise, we can pick the next largest transaction. I think
> > we can retry it for few times and if we get the incomplete data for
> > multiple transactions, then we can decide to spill the transaction or
> > maybe we can directly spill the first largest transaction which has
> > incomplete data.
> >
> Yeah, we might do something on this line. Basically, we need to mark
> the top-transaction as data-incomplete if any of its subtransaction is
> having data-incomplete (it will always be the latest sub-transaction
> of the top transaction). Also, for streaming, we are checking the
> largest top transaction whereas for spilling we just need the larget
> (sub) transaction. So we also need to decide while picking the
> largest top transaction for streaming, if we get a few transactions
> with in-complete data then how we will go for the spill. Do we spill
> all the sub-transactions under this top transaction or we will again
> find the larget (sub) transaction for spilling.
>
I think it is better to do later as that will lead to the spill of
only required (minimum changes to get the memory below threshold)
changes.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-09 07:10 ` Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-01-09 07:10 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 9, 2020 at 12:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jan 9, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > I have observed one more design issue.
> > > >
> > >
> > > Good observation.
> > >
> > > > The problem is that when we
> > > > get a toasted chunks we remember the changes in the memory(hash table)
> > > > but don't stream until we get the actual change on the main table.
> > > > Now, the problem is that we might get the change of the toasted table
> > > > and the main table in different streams. So basically, in a stream,
> > > > if we have only got the toasted tuples then even after
> > > > ReorderBufferStreamTXN the memory usage will not be reduced.
> > > >
> > >
> > > I think we can't split such changes in a different stream (unless we
> > > design an entirely new solution to send partial changes of toast
> > > data), so we need to send them together. We can keep a flag like
> > > data_complete in ReorderBufferTxn and mark it complete only when we
> > > are able to assemble the entire tuple. Now, whenever, we try to
> > > stream the changes once we reach the memory threshold, we can check
> > > whether the data_complete flag is true, if so, then only send the
> > > changes, otherwise, we can pick the next largest transaction. I think
> > > we can retry it for few times and if we get the incomplete data for
> > > multiple transactions, then we can decide to spill the transaction or
> > > maybe we can directly spill the first largest transaction which has
> > > incomplete data.
> > >
> > Yeah, we might do something on this line. Basically, we need to mark
> > the top-transaction as data-incomplete if any of its subtransaction is
> > having data-incomplete (it will always be the latest sub-transaction
> > of the top transaction). Also, for streaming, we are checking the
> > largest top transaction whereas for spilling we just need the larget
> > (sub) transaction. So we also need to decide while picking the
> > largest top transaction for streaming, if we get a few transactions
> > with in-complete data then how we will go for the spill. Do we spill
> > all the sub-transactions under this top transaction or we will again
> > find the larget (sub) transaction for spilling.
> >
>
> I think it is better to do later as that will lead to the spill of
> only required (minimum changes to get the memory below threshold)
> changes.
Make sense to me.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-13 09:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-13 09:48 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 9, 2020 at 12:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jan 9, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Jan 8, 2020 at 1:12 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > I have observed one more design issue.
> > > >
> > >
> > > Good observation.
> > >
> > > > The problem is that when we
> > > > get a toasted chunks we remember the changes in the memory(hash table)
> > > > but don't stream until we get the actual change on the main table.
> > > > Now, the problem is that we might get the change of the toasted table
> > > > and the main table in different streams. So basically, in a stream,
> > > > if we have only got the toasted tuples then even after
> > > > ReorderBufferStreamTXN the memory usage will not be reduced.
> > > >
> > >
> > > I think we can't split such changes in a different stream (unless we
> > > design an entirely new solution to send partial changes of toast
> > > data), so we need to send them together. We can keep a flag like
> > > data_complete in ReorderBufferTxn and mark it complete only when we
> > > are able to assemble the entire tuple. Now, whenever, we try to
> > > stream the changes once we reach the memory threshold, we can check
> > > whether the data_complete flag is true, if so, then only send the
> > > changes, otherwise, we can pick the next largest transaction. I think
> > > we can retry it for few times and if we get the incomplete data for
> > > multiple transactions, then we can decide to spill the transaction or
> > > maybe we can directly spill the first largest transaction which has
> > > incomplete data.
> > >
> > Yeah, we might do something on this line. Basically, we need to mark
> > the top-transaction as data-incomplete if any of its subtransaction is
> > having data-incomplete (it will always be the latest sub-transaction
> > of the top transaction). Also, for streaming, we are checking the
> > largest top transaction whereas for spilling we just need the larget
> > (sub) transaction. So we also need to decide while picking the
> > largest top transaction for streaming, if we get a few transactions
> > with in-complete data then how we will go for the spill. Do we spill
> > all the sub-transactions under this top transaction or we will again
> > find the larget (sub) transaction for spilling.
> >
>
> I think it is better to do later as that will lead to the spill of
> only required (minimum changes to get the memory below threshold)
> changes.
I think instead of doing this can't we just spill the changes which
are in toast_hash. Basically, at the end of the stream, we have some
toast tuple which we could not stream because we did not have the
insert for the main table then we can spill only those changes which
are in tuple hash. And, in the subsequent stream whenever we get the
insert for the main table at that time we can restore those changes
and stream. We can also maintain some flag saying data is not
complete (with some change LSN number) and after that LSN we can spill
any toast change to disk until we get the change for the main table,
by this way we can avoid building tuple hash until we get the change
for the main table.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-14 05:13 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-14 05:13 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 13, 2020 at 3:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Jan 9, 2020 at 12:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Jan 9, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > > The problem is that when we
> > > > > get a toasted chunks we remember the changes in the memory(hash table)
> > > > > but don't stream until we get the actual change on the main table.
> > > > > Now, the problem is that we might get the change of the toasted table
> > > > > and the main table in different streams. So basically, in a stream,
> > > > > if we have only got the toasted tuples then even after
> > > > > ReorderBufferStreamTXN the memory usage will not be reduced.
> > > > >
> > > >
> > > > I think we can't split such changes in a different stream (unless we
> > > > design an entirely new solution to send partial changes of toast
> > > > data), so we need to send them together. We can keep a flag like
> > > > data_complete in ReorderBufferTxn and mark it complete only when we
> > > > are able to assemble the entire tuple. Now, whenever, we try to
> > > > stream the changes once we reach the memory threshold, we can check
> > > > whether the data_complete flag is true
Here, we can also consider streaming the changes when data_complete is
false, but some additional changes have been added to the same txn as
the new changes might complete the tuple.
> > > > , if so, then only send the
> > > > changes, otherwise, we can pick the next largest transaction. I think
> > > > we can retry it for few times and if we get the incomplete data for
> > > > multiple transactions, then we can decide to spill the transaction or
> > > > maybe we can directly spill the first largest transaction which has
> > > > incomplete data.
> > > >
> > > Yeah, we might do something on this line. Basically, we need to mark
> > > the top-transaction as data-incomplete if any of its subtransaction is
> > > having data-incomplete (it will always be the latest sub-transaction
> > > of the top transaction). Also, for streaming, we are checking the
> > > largest top transaction whereas for spilling we just need the larget
> > > (sub) transaction. So we also need to decide while picking the
> > > largest top transaction for streaming, if we get a few transactions
> > > with in-complete data then how we will go for the spill. Do we spill
> > > all the sub-transactions under this top transaction or we will again
> > > find the larget (sub) transaction for spilling.
> > >
> >
> > I think it is better to do later as that will lead to the spill of
> > only required (minimum changes to get the memory below threshold)
> > changes.
> I think instead of doing this can't we just spill the changes which
> are in toast_hash. Basically, at the end of the stream, we have some
> toast tuple which we could not stream because we did not have the
> insert for the main table then we can spill only those changes which
> are in tuple hash.
>
Hmm, I think this can turn out to be inefficient because we can easily
end up spilling the data even when we don't need to so. Consider
cases, where part of the streamed changes are for toast, and remaining
are the changes which we would have streamed and hence can be removed.
In such cases, we could have easily consumed remaining changes for
toast without spilling. Also, I am not sure if spilling changes from
the hash table is a good idea as they are no more in the same order as
they were in ReorderBuffer which means the order in which we serialize
the changes normally would change and that might have some impact, so
we would need some more study if we want to pursue this idea.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-22 05:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-22 05:00 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 13, 2020 at 3:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Jan 9, 2020 at 12:09 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Jan 9, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > On Thu, Jan 9, 2020 at 9:35 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > > > The problem is that when we
> > > > > > get a toasted chunks we remember the changes in the memory(hash table)
> > > > > > but don't stream until we get the actual change on the main table.
> > > > > > Now, the problem is that we might get the change of the toasted table
> > > > > > and the main table in different streams. So basically, in a stream,
> > > > > > if we have only got the toasted tuples then even after
> > > > > > ReorderBufferStreamTXN the memory usage will not be reduced.
> > > > > >
> > > > >
> > > > > I think we can't split such changes in a different stream (unless we
> > > > > design an entirely new solution to send partial changes of toast
> > > > > data), so we need to send them together. We can keep a flag like
> > > > > data_complete in ReorderBufferTxn and mark it complete only when we
> > > > > are able to assemble the entire tuple. Now, whenever, we try to
> > > > > stream the changes once we reach the memory threshold, we can check
> > > > > whether the data_complete flag is true
>
> Here, we can also consider streaming the changes when data_complete is
> false, but some additional changes have been added to the same txn as
> the new changes might complete the tuple.
>
> > > > > , if so, then only send the
> > > > > changes, otherwise, we can pick the next largest transaction. I think
> > > > > we can retry it for few times and if we get the incomplete data for
> > > > > multiple transactions, then we can decide to spill the transaction or
> > > > > maybe we can directly spill the first largest transaction which has
> > > > > incomplete data.
> > > > >
> > > > Yeah, we might do something on this line. Basically, we need to mark
> > > > the top-transaction as data-incomplete if any of its subtransaction is
> > > > having data-incomplete (it will always be the latest sub-transaction
> > > > of the top transaction). Also, for streaming, we are checking the
> > > > largest top transaction whereas for spilling we just need the larget
> > > > (sub) transaction. So we also need to decide while picking the
> > > > largest top transaction for streaming, if we get a few transactions
> > > > with in-complete data then how we will go for the spill. Do we spill
> > > > all the sub-transactions under this top transaction or we will again
> > > > find the larget (sub) transaction for spilling.
> > > >
> > >
> > > I think it is better to do later as that will lead to the spill of
> > > only required (minimum changes to get the memory below threshold)
> > > changes.
> > I think instead of doing this can't we just spill the changes which
> > are in toast_hash. Basically, at the end of the stream, we have some
> > toast tuple which we could not stream because we did not have the
> > insert for the main table then we can spill only those changes which
> > are in tuple hash.
> >
>
> Hmm, I think this can turn out to be inefficient because we can easily
> end up spilling the data even when we don't need to so. Consider
> cases, where part of the streamed changes are for toast, and remaining
> are the changes which we would have streamed and hence can be removed.
> In such cases, we could have easily consumed remaining changes for
> toast without spilling. Also, I am not sure if spilling changes from
> the hash table is a good idea as they are no more in the same order as
> they were in ReorderBuffer which means the order in which we serialize
> the changes normally would change and that might have some impact, so
> we would need some more study if we want to pursue this idea.
I have fixed this bug and attached it as a separate patch. I will
merge it to the main patch after we agree with the idea and after some
more testing.
The idea is that whenever we get the toasted chunk instead of directly
inserting it into the toast hash I am inserting it into some local
list so that if we don't get the change for the main table then we can
insert these changes back to the txn->changes. So once we get the
change for the main table at that time I am preparing the hash table
to merge the chunks. If the stream is over and we haven't got the
changes for the main table, that time we will mark the txn that it has
some pending toast changes so that next time we will not pick the same
transaction for the streaming. This flag will be cleaned whenever we
get any changes for the txn (insert or /update). There is also a
possibility that even after we stream the changes the rb->size is not
below logical_decoding_work_mem because we could not stream the
changes so for handling this after streaming we recheck the size and
if it is still not under control then we pick another transaction. In
some cases, we might not get any transaction to stream because the
transaction has the pending toast change flag set, In this case, we
will go for the spill.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v8-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.3K) download
From 003bd22dd11b032cbfecb7cd8785ce5ecbaa60e5 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v8 01/13] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 017f03b..51557e2 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 2fa0a7f..b11b0c2 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 3aa6812..8c281e8 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 5e1dc8a..a99fcaf 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 98b033f..e23892a 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index f7cc8c4..4805cae 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v8-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From e8bbd48b3d309041b1205e46aa697ac0c72c5f46 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v8 03/13] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index cd105d9..a3efbfd 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..ace21ec 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index bdf4389..9c95fc1 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 9a3f045..15bb5ed 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -356,6 +356,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -395,6 +441,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v8-0002-Issue-individual-invalidations-with-wal_level-log.patch (application/octet-stream, 16.4K) download
From 65adaeebc3127ae732a0927e22768594d0ee0d12 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v8 02/13] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 50 +++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 ++++++++
src/backend/replication/logical/reorderbuffer.c | 55 +++++++++++++++---
src/backend/utils/cache/inval.c | 75 +++++++++++++++++++++++++
src/include/access/xact.h | 18 +++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 234 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index e388cc7..6e46d19 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,44 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 51557e2..dd3d36f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a99fcaf..13a11ac 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 73ca4f7..78443e2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -473,6 +473,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1822,17 +1823,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2212,6 +2218,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2658,6 +2696,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2765,6 +2804,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3050,6 +3090,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..e0d04b9 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write individual invalidations into WAL to support
+ * the decoding of the in-progress transaction. As of now it was enough to
+ * log invalidation only at commit because we are only decoding the transaction
+ * at the commit time. We only need to log the catalog cache and relcache
+ * invalidation. There can not be any active MVCC scan in logical decoding so
+ * we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +499,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +523,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +545,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +565,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1552,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..6f2a583 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 626ecf4..9a3f045 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -459,6 +470,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v8-0004-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (application/octet-stream, 13.1K) download
From 3134014b6fe09daafbbc52b39dead9332683f4b0 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v8 04/13] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++
src/backend/access/index/genam.c | 49 +++++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++--
src/backend/utils/time/snapmgr.c | 25 ++++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 124 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index ace21ec..319349a 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 5ddb6e8..b6c95c3 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1431,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1552,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1708,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5514,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index c16eb05..5644b8d 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,22 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +530,22 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +672,22 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 78443e2..e7aa004 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -692,7 +692,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1551,7 +1551,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1802,7 +1802,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +1822,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v8-0005-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 37.2K) download
From 30d5fdb4efcae52cd00eee4ea979d9c5a22e7b82 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v8 05/13] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 692 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 36 ++
3 files changed, 673 insertions(+), 93 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index e7aa004..c77aabe 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -769,6 +782,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -864,6 +909,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -987,7 +1035,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1023,6 +1071,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1037,6 +1088,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1320,6 +1374,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1345,8 +1408,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1354,9 +1502,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1495,63 +1640,48 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * build data to be able to lookup the CommandIds of catalog tuples
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
-
- snapshot_now = txn->base_snapshot;
-
- /* build data to be able to lookup the CommandIds of catalog tuples */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1567,15 +1697,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1583,6 +1718,35 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ if (streaming)
+ {
+ /*
+ * While streaming an in-progress transaction there is a
+ * possibility that the (sub)transaction might get aborted
+ * concurrently. In such case if the (sub)transaction has
+ * catalog update then we might decode the tuple using wrong
+ * catalog version. So for detecting the concurrent abort we
+ * set CheckXidAlive to the current (sub)transaction's xid for
+ * which this change belongs to. And, during catalog scan we
+ * can check the status of the xid and if it is aborted we will
+ * report an specific error which we can ignore. We might have
+ * already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the
+ * abort we will stream abort message to truncate the changes in
+ * the subscriber.
+ */
+ CheckXidAlive = change->txn->xid;
+ }
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1592,8 +1756,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1659,7 +1821,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1680,8 +1850,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1699,7 +1867,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1757,7 +1925,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1766,10 +1942,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1800,9 +1982,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +2004,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1860,14 +2043,41 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ if (!XLogRecPtrIsInvalid(prev_lsn))
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1885,14 +2095,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then Discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,18 +2129,117 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1946,6 +2263,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2015,6 +2339,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2150,8 +2481,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2159,6 +2499,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2170,19 +2511,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2211,6 +2561,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2300,6 +2651,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2404,6 +2762,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2423,15 +2813,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2734,6 +3155,101 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after we there
+ * might be some new sub-transaction which after the last streaming run
+ * so we need to add those sub-xip in the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 15bb5ed..adb8f9d 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -192,6 +193,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -227,6 +246,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -257,6 +286,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v8-0006-Fix-speculative-insert-bug.patch (application/octet-stream, 2.5K) download
From d26729956f5ff279b7c25f4ec1f2b671702c7df7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 10 Jan 2020 09:01:35 +0530
Subject: [PATCH v8 06/13] Fix speculative insert bug.
---
src/backend/replication/logical/reorderbuffer.c | 23 +++++++++++++++++++----
src/include/replication/reorderbuffer.h | 6 ++++++
2 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c77aabe..070ad1f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1701,6 +1701,16 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
+ /*
+ * Resotre any previous speculative inserted tuple if we are running in
+ * streaming mode.
+ */
+ if (streaming && txn->specinsert != NULL)
+ {
+ specinsert = txn->specinsert;
+ txn->specinsert = NULL;
+ }
+
if (using_subtxn)
BeginInternalSubTransaction("stream");
else
@@ -2029,13 +2039,18 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
- * There's a speculative insertion remaining, just clean in up, it
- * can't have been successful, otherwise we'd gotten a confirmation
- * record.
+ * In non-streaming mode if there's a speculative insertion remaining,
+ * just clean in up, it can't have been successful, otherwise we'd
+ * gotten a confirmation record. For streaming mode, remember the tuple
+ * so that if we get the confirmation in the next stream we can stream
+ * it.
*/
if (specinsert)
{
- ReorderBufferReturnChange(rb, specinsert);
+ if (streaming)
+ txn->specinsert = specinsert;
+ else
+ ReorderBufferReturnChange(rb, specinsert);
specinsert = NULL;
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index adb8f9d..2680ac3 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -344,6 +344,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Speculative insert saved from the last streamed run if the speculative
+ * confirm has not received in the same stream.
+ */
+ ReorderBufferChange *specinsert;
+
/* ---
* Position in one of three lists:
* * list of subtransactions if we are *known* to be subxact
--
1.8.3.1
v8-0007-Support-logical_decoding_work_mem-set-from-create.patch (application/octet-stream, 13.1K) download
From c85c9bf20abf9c5f4037cb5fdbacfc96fea2cb11 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v8 07/13] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e07dc01..6d1a25b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index f77a83b..5cd1daa 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9bfe142..c50e854 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -668,10 +689,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -696,6 +720,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -707,7 +738,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -745,7 +777,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -782,7 +814,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9..896ddab 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -408,6 +408,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7a5471f..48b960c 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1745,6 +1745,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 7525082..536722b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..3394379 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e08afc6..4c7acfb 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -169,6 +169,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
v8-0009-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From 99d8c2c35690e2e802536a116e2f528ebd49bcd3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Jan 2020 09:45:27 +0530
Subject: [PATCH v8 09/13] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 0bfd615..512d843 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2004,6 +2004,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index c9e75f4..e406856 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -785,7 +785,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 070ad1f..fe4e57c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3260,6 +3264,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 63fc2c7..d7f22ae 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1293,7 +1293,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1314,7 +1314,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2357,6 +2358,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3196,7 +3200,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3253,6 +3257,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3276,6 +3283,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3362,6 +3372,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3610,11 +3625,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fcf2a12..75124c8 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5173,9 +5173,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 2680ac3..02650c3 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -525,15 +525,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f..3888b0c 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 70e1e2f..9dc739e 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1982,9 +1982,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v8-0010-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From dd06dab3c58def3943d2622047cd03b76cc467ca Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v8 10/13] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 6da7f71..086d0c7 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -70,7 +70,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v8-0008-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 91.1K) download
From 574b34383c118b0921000e5ae4db8d73238ed691 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 13 Jan 2020 14:24:39 +0530
Subject: [PATCH v8 08/13] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 310 +++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2074 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 5cd1daa..1dc486c 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index c50e854..a486bd3 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -691,11 +725,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -727,6 +764,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -739,7 +783,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -777,7 +821,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -814,7 +859,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 51c486b..03ef76c 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 896ddab..1b8303c 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -408,8 +408,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..e80d00c 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 9c95fc1..61064f3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd..93780b2 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,10 +139,15 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -178,8 +183,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -187,6 +192,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -248,13 +257,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -296,6 +310,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -305,6 +320,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -347,12 +366,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -397,7 +420,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -405,6 +428,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -685,3 +712,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 48b960c..5c20c0e 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -529,6 +636,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found PG_USED_FOR_ASSERTS_ONLY = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -695,6 +1123,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -830,6 +1261,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -929,6 +1363,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1020,6 +1457,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1117,6 +1570,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1132,6 +1601,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1580,6 +2052,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1746,6 +2776,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 536722b..ebe0423 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,17 +45,45 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -64,11 +92,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -84,16 +118,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +206,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +235,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +259,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +280,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +368,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +435,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +463,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +495,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +512,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +521,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +547,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +571,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +652,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -549,6 +773,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*/
static RelationSyncEntry *
@@ -623,6 +875,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -657,7 +939,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index bb69683..3085c0f 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 9c06374..63fc2c7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -969,6 +969,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3394379..18f416f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index aecb601..146d7c4 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2cc2dc4..ade4188 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 4c7acfb..54054a4 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -170,6 +170,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v8-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1013B) download
From bb44ee6ad6ff0c99a8f759fa7efa5b242bd19898 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v8 11/13] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index fe4e57c..beb6cd2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1327,6 +1327,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
v8-0012-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From e32d888e9b1de0c25843eae0d299b7f9e8dc45fb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v8 12/13] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v8-0013-Bugfix-handling-of-incomplete-toast-tuple.patch (application/octet-stream, 10.2K) download
From 4fd60ba2487e6f57d435cc57449ad2a8b5287632 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Tue, 21 Jan 2020 10:57:10 +0530
Subject: [PATCH v8 13/13] Bugfix handling of incomplete toast tuple
---
contrib/test_decoding/logical.conf | 1 +
src/backend/replication/logical/reorderbuffer.c | 159 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 8 ++
3 files changed, 152 insertions(+), 16 deletions(-)
diff --git a/contrib/test_decoding/logical.conf b/contrib/test_decoding/logical.conf
index 07c4d3d..f748994 100644
--- a/contrib/test_decoding/logical.conf
+++ b/contrib/test_decoding/logical.conf
@@ -1,3 +1,4 @@
wal_level = logical
max_replication_slots = 4
logical_decoding_work_mem = 64kB
+logging_collector=on
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index beb6cd2..acede78 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -664,6 +664,16 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If we have detected the toast chunk without the main table change while
+ * sending the previous stream, then reset the flag if we have got any
+ * insert/update so that we can retry to stream this.
+ */
+ if (txn->incomplte_toast_chunks &&
+ (change->action == REORDER_BUFFER_CHANGE_INSERT ||
+ change->action == REORDER_BUFFER_CHANGE_UPDATE))
+ txn->incomplte_toast_chunks = false;
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, true);
@@ -1682,6 +1692,8 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferIterTXNState *volatile iterstate = NULL;
volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ dlist_head toast_change;
+ Oid toastrelid = InvalidOid;
/*
* build data to be able to lookup the CommandIds of catalog tuples
@@ -1691,6 +1703,9 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* setup the initial snapshot */
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
+ /* Initialize the local toast change list. */
+ dlist_init(&toast_change);
+
/*
* Decoding needs access to syscaches et al., which in turn use
* heavyweight locks and such. Thus we need to have enough state around to
@@ -1734,6 +1749,7 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
{
Relation relation = NULL;
+ Relation toastrel = NULL;
Oid reloid;
/*
@@ -1838,6 +1854,32 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* user-triggered change */
if (!IsToastRelation(relation))
{
+ /*
+ * If we have the toast changes list then reassemble the
+ * toast chunks in the hash table.
+ */
+ if (!dlist_is_empty(&toast_change))
+ {
+ dlist_mutable_iter iter;
+
+ Assert(streaming);
+
+ /* open the toast relation. */
+ toastrel = RelationIdGetRelation(toastrelid);
+
+ dlist_foreach_modify(iter, &toast_change)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange,
+ node, iter.cur);
+ dlist_delete(&change->node);
+ ReorderBufferToastAppendChunk(rb, txn, toastrel,
+ change);
+ }
+ RelationClose(toastrel);
+ }
+
ReorderBufferToastReplace(rb, txn, relation, change);
if (streaming)
{
@@ -1869,8 +1911,25 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* disk.
*/
dlist_delete(&change->node);
- ReorderBufferToastAppendChunk(rb, txn, relation,
- change);
+
+ /*
+ * For the streaming run don't directly reassemble the
+ * chunks in the hash table instead collect those
+ * changes in the list because there is a possibility
+ * that we might not get the change for the main table
+ * in this stream. So we will assemble the chunks when
+ * we actually get the change for the main table.
+ * Otherwise we will attach the list back to the main
+ * changes list.
+ */
+ if (streaming)
+ {
+ dlist_push_tail(&toast_change, &change->node);
+ toastrelid = reloid;
+ }
+ else
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
}
change_done:
@@ -2125,7 +2184,36 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* deallocate the ReorderBufferTXN.
*/
if (streaming)
+ {
ReorderBufferTruncateTXN(rb, txn);
+
+ /*
+ * If we could not stream the toasted chunks then append them back
+ * to the main txn list.
+ */
+ if (!dlist_is_empty(&toast_change))
+ {
+ dlist_mutable_iter iter;
+
+ Assert(streaming);
+ dlist_foreach_modify(iter, &toast_change)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /*
+ * Remove from temp list and add it back to the txn changes
+ * list.
+ */
+ dlist_delete(&change->node);
+ dlist_push_tail(&txn->changes, &change->node);
+ txn->nentries_mem++;
+ txn->nentries++;
+ }
+ txn->incomplte_toast_chunks = true;
+ }
+ }
else
ReorderBufferCleanupTXN(rb, txn);
}
@@ -2177,6 +2265,33 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
rb->stream_stop(rb, txn);
FlushErrorState();
+
+ /*
+ * If we could not stream the toasted chunks then append them
+ * back to the main txn list.
+ */
+ if (!dlist_is_empty(&toast_change))
+ {
+ dlist_mutable_iter iter;
+
+ Assert(streaming);
+ dlist_foreach_modify(iter, &toast_change)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /*
+ * Remove from temp list and add it back to the txn
+ * changes list.
+ */
+ dlist_delete(&change->node);
+ dlist_push_tail(&txn->changes, &change->node);
+ txn->nentries_mem++;
+ txn->nentries++;
+ }
+ txn->incomplte_toast_chunks = true;
+ }
}
}
else
@@ -2523,6 +2638,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
Size sz;
ReorderBufferTXN *txn;
+ ReorderBufferTXN *toptxn = NULL;
Assert(change->txn);
@@ -2538,7 +2654,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
/* if subxact, and streaming supported, use the toplevel instead */
if (txn->toptxn && ReorderBufferCanStream(rb))
- txn = txn->toptxn;
+ toptxn = txn->toptxn;
sz = ReorderBufferChangeSize(change);
@@ -2546,12 +2662,16 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
txn->size += sz;
rb->size += sz;
+ if (toptxn)
+ toptxn->size += sz;
}
else
{
Assert((rb->size >= sz) && (txn->size >= sz));
txn->size -= sz;
rb->size -= sz;
+ if (toptxn)
+ toptxn->size -= sz;
}
Assert(txn->size <= rb->size);
@@ -2805,14 +2925,11 @@ ReorderBufferLargestTopTXN(ReorderBuffer *rb)
txn = dlist_container(ReorderBufferTXN, node, iter.cur);
/* if the current transaction is larger, remember it */
- if ((!largest) || (txn->size > largest->size))
- largest = txn;
+ if (((!largest) || (txn->size > largest->size)) &&
+ (!txn->incomplte_toast_chunks) && (txn->size > 0))
+ largest = txn;
}
- Assert(largest);
- Assert(largest->size > 0);
- Assert(largest->size <= rb->size);
-
return largest;
}
@@ -2834,11 +2951,13 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
if (rb->size < logical_decoding_work_mem * 1024L)
return;
+retry:
/*
* Pick the largest transaction (or subtransaction) and evict it from
* memory by streaming, if supported. Otherwise spill to disk.
*/
- if (ReorderBufferCanStream(rb))
+ if (ReorderBufferCanStream(rb) &&
+ (txn = ReorderBufferLargestTopTXN(rb)) != NULL)
{
/*
* Pick the largest toplevel transaction and evict it from memory by
@@ -2877,8 +2996,8 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
* streaming. But for streaming we should really check nentries_mem for
* all subtransactions too.
*/
- Assert(txn->size == 0);
- Assert(txn->nentries_mem == 0);
+ Assert(txn->incomplte_toast_chunks || txn->specinsert || txn->size == 0);
+ Assert(txn->incomplte_toast_chunks || txn->nentries_mem == 0);
/*
* And furthermore, evicting the transaction should get us below the
@@ -2890,7 +3009,16 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
* the memory limit). So by evicting it we're definitely back below the
* memory limit.
*/
- Assert(rb->size < logical_decoding_work_mem * 1024L);
+ Assert(txn->incomplte_toast_chunks ||
+ rb->size < logical_decoding_work_mem * 1024L);
+
+ /*
+ * For streaming transaction it's possible that we are not yet below the
+ * memory limit due to incomplete toast tuple. So we retry with some other
+ * transaction.
+ */
+ if (rb->size >= logical_decoding_work_mem * 1024L)
+ goto retry;
}
/*
@@ -3277,9 +3405,8 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* Don't consider already streamed transaction. */
rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
- Assert(dlist_is_empty(&txn->changes));
- Assert(txn->nentries == 0);
- Assert(txn->nentries_mem == 0);
+ Assert(txn->incomplte_toast_chunks || txn->nentries == 0);
+ Assert(txn->incomplte_toast_chunks || txn->nentries_mem == 0);
}
/*
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 02650c3..1348cde 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -251,6 +251,14 @@ typedef struct ReorderBufferTXN
bool any_data_sent;
/*
+ * While sending the last stream we have found that the transaction has some
+ * toast tuples but haven't yet got the change of the main table so we could
+ * not stream it. Don't try to stream it again until we get any new change
+ * for the transaction.
+ */
+ bool incomplte_toast_chunks;
+
+ /*
* Toplevel transaction for this subxact (NULL for top-level).
*/
struct ReorderBufferTXN *toptxn;
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-28 05:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-28 05:57 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Jan 22, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > Hmm, I think this can turn out to be inefficient because we can easily
> > end up spilling the data even when we don't need to so. Consider
> > cases, where part of the streamed changes are for toast, and remaining
> > are the changes which we would have streamed and hence can be removed.
> > In such cases, we could have easily consumed remaining changes for
> > toast without spilling. Also, I am not sure if spilling changes from
> > the hash table is a good idea as they are no more in the same order as
> > they were in ReorderBuffer which means the order in which we serialize
> > the changes normally would change and that might have some impact, so
> > we would need some more study if we want to pursue this idea.
> I have fixed this bug and attached it as a separate patch. I will
> merge it to the main patch after we agree with the idea and after some
> more testing.
>
> The idea is that whenever we get the toasted chunk instead of directly
> inserting it into the toast hash I am inserting it into some local
> list so that if we don't get the change for the main table then we can
> insert these changes back to the txn->changes. So once we get the
> change for the main table at that time I am preparing the hash table
> to merge the chunks.
>
I think this idea will work but appears to be quite costly because (a)
you might need to serialize/deserialize the changes multiple times and
might attempt streaming multiple times even though you can't do (b)
you need to remove/add the same set of changes from the main list
multiple times.
It seems to me that we need to add all of this new handling because
while taking the decision whether to stream or not we don't know
whether the txn has changes that can't be streamed. One idea to make
it work is that we identify it while decoding the WAL. I think we
need to set a bit in the insert/delete WAL record to identify if the
tuple belongs to a toast relation. This won't add any additional
overhead in WAL and reduce a lot of complexity in the logical decoding
and also decoding will be efficient. If this is feasible, then we can
do the same for speculative insertions.
In patch v8-0013-Bugfix-handling-of-incomplete-toast-tuple, why is
below change required?
--- a/contrib/test_decoding/logical.conf
+++ b/contrib/test_decoding/logical.conf
@@ -1,3 +1,4 @@
wal_level = logical
max_replication_slots = 4
logical_decoding_work_mem = 64kB
+logging_collector=on
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-28 06:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 06:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-28 06:04 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 11:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Jan 22, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > Hmm, I think this can turn out to be inefficient because we can easily
> > > end up spilling the data even when we don't need to so. Consider
> > > cases, where part of the streamed changes are for toast, and remaining
> > > are the changes which we would have streamed and hence can be removed.
> > > In such cases, we could have easily consumed remaining changes for
> > > toast without spilling. Also, I am not sure if spilling changes from
> > > the hash table is a good idea as they are no more in the same order as
> > > they were in ReorderBuffer which means the order in which we serialize
> > > the changes normally would change and that might have some impact, so
> > > we would need some more study if we want to pursue this idea.
> > I have fixed this bug and attached it as a separate patch. I will
> > merge it to the main patch after we agree with the idea and after some
> > more testing.
> >
> > The idea is that whenever we get the toasted chunk instead of directly
> > inserting it into the toast hash I am inserting it into some local
> > list so that if we don't get the change for the main table then we can
> > insert these changes back to the txn->changes. So once we get the
> > change for the main table at that time I am preparing the hash table
> > to merge the chunks.
> >
>
>
> I think this idea will work but appears to be quite costly because (a)
> you might need to serialize/deserialize the changes multiple times and
> might attempt streaming multiple times even though you can't do (b)
> you need to remove/add the same set of changes from the main list
> multiple times.
I agree with this.
>
> It seems to me that we need to add all of this new handling because
> while taking the decision whether to stream or not we don't know
> whether the txn has changes that can't be streamed. One idea to make
> it work is that we identify it while decoding the WAL. I think we
> need to set a bit in the insert/delete WAL record to identify if the
> tuple belongs to a toast relation. This won't add any additional
> overhead in WAL and reduce a lot of complexity in the logical decoding
> and also decoding will be efficient. If this is feasible, then we can
> do the same for speculative insertions.
The Idea looks good to me. I will work on this.
>
> In patch v8-0013-Bugfix-handling-of-incomplete-toast-tuple, why is
> below change required?
>
> --- a/contrib/test_decoding/logical.conf
> +++ b/contrib/test_decoding/logical.conf
> @@ -1,3 +1,4 @@
> wal_level = logical
> max_replication_slots = 4
> logical_decoding_work_mem = 64kB
> +logging_collector=on
Sorry, these are some local changes which got included in the patch.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-28 06:12 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-04 05:29 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-28 06:12 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Jan 22, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > >
> > > > Hmm, I think this can turn out to be inefficient because we can easily
> > > > end up spilling the data even when we don't need to so. Consider
> > > > cases, where part of the streamed changes are for toast, and remaining
> > > > are the changes which we would have streamed and hence can be removed.
> > > > In such cases, we could have easily consumed remaining changes for
> > > > toast without spilling. Also, I am not sure if spilling changes from
> > > > the hash table is a good idea as they are no more in the same order as
> > > > they were in ReorderBuffer which means the order in which we serialize
> > > > the changes normally would change and that might have some impact, so
> > > > we would need some more study if we want to pursue this idea.
> > > I have fixed this bug and attached it as a separate patch. I will
> > > merge it to the main patch after we agree with the idea and after some
> > > more testing.
> > >
> > > The idea is that whenever we get the toasted chunk instead of directly
> > > inserting it into the toast hash I am inserting it into some local
> > > list so that if we don't get the change for the main table then we can
> > > insert these changes back to the txn->changes. So once we get the
> > > change for the main table at that time I am preparing the hash table
> > > to merge the chunks.
> > >
> >
> >
> > I think this idea will work but appears to be quite costly because (a)
> > you might need to serialize/deserialize the changes multiple times and
> > might attempt streaming multiple times even though you can't do (b)
> > you need to remove/add the same set of changes from the main list
> > multiple times.
> I agree with this.
> >
> > It seems to me that we need to add all of this new handling because
> > while taking the decision whether to stream or not we don't know
> > whether the txn has changes that can't be streamed. One idea to make
> > it work is that we identify it while decoding the WAL. I think we
> > need to set a bit in the insert/delete WAL record to identify if the
> > tuple belongs to a toast relation. This won't add any additional
> > overhead in WAL and reduce a lot of complexity in the logical decoding
> > and also decoding will be efficient. If this is feasible, then we can
> > do the same for speculative insertions.
> The Idea looks good to me. I will work on this.
>
One more thing we can do is to identify whether the tuple belongs to
toast relation while decoding it. However, I think to do that we need
to have access to relcache at that time and that might add some
overhead as we need to do that for each tuple. Can we investigate
what it will take to do that and if it is better than setting a bit
during WAL logging.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 06:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-28 06:28 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 08:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-28 06:28 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2020 at 11:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Jan 22, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Hmm, I think this can turn out to be inefficient because we can easily
> > > > > end up spilling the data even when we don't need to so. Consider
> > > > > cases, where part of the streamed changes are for toast, and remaining
> > > > > are the changes which we would have streamed and hence can be removed.
> > > > > In such cases, we could have easily consumed remaining changes for
> > > > > toast without spilling. Also, I am not sure if spilling changes from
> > > > > the hash table is a good idea as they are no more in the same order as
> > > > > they were in ReorderBuffer which means the order in which we serialize
> > > > > the changes normally would change and that might have some impact, so
> > > > > we would need some more study if we want to pursue this idea.
> > > > I have fixed this bug and attached it as a separate patch. I will
> > > > merge it to the main patch after we agree with the idea and after some
> > > > more testing.
> > > >
> > > > The idea is that whenever we get the toasted chunk instead of directly
> > > > inserting it into the toast hash I am inserting it into some local
> > > > list so that if we don't get the change for the main table then we can
> > > > insert these changes back to the txn->changes. So once we get the
> > > > change for the main table at that time I am preparing the hash table
> > > > to merge the chunks.
> > > >
> > >
> > >
> > > I think this idea will work but appears to be quite costly because (a)
> > > you might need to serialize/deserialize the changes multiple times and
> > > might attempt streaming multiple times even though you can't do (b)
> > > you need to remove/add the same set of changes from the main list
> > > multiple times.
> > I agree with this.
> > >
> > > It seems to me that we need to add all of this new handling because
> > > while taking the decision whether to stream or not we don't know
> > > whether the txn has changes that can't be streamed. One idea to make
> > > it work is that we identify it while decoding the WAL. I think we
> > > need to set a bit in the insert/delete WAL record to identify if the
> > > tuple belongs to a toast relation. This won't add any additional
> > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > and also decoding will be efficient. If this is feasible, then we can
> > > do the same for speculative insertions.
> > The Idea looks good to me. I will work on this.
> >
>
> One more thing we can do is to identify whether the tuple belongs to
> toast relation while decoding it. However, I think to do that we need
> to have access to relcache at that time and that might add some
> overhead as we need to do that for each tuple. Can we investigate
> what it will take to do that and if it is better than setting a bit
> during WAL logging.
IMHO, for the catalog scan, we will have to start/stop the transaction
for each change. So do you want that we should evaluate its
performance? Also, during we get the change we might not have the
complete historic snapshot ready to fetch the rel cache entry.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 06:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-28 08:00 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 08:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-28 08:00 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 11:58 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > > > It seems to me that we need to add all of this new handling because
> > > > while taking the decision whether to stream or not we don't know
> > > > whether the txn has changes that can't be streamed. One idea to make
> > > > it work is that we identify it while decoding the WAL. I think we
> > > > need to set a bit in the insert/delete WAL record to identify if the
> > > > tuple belongs to a toast relation. This won't add any additional
> > > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > > and also decoding will be efficient. If this is feasible, then we can
> > > > do the same for speculative insertions.
> > > The Idea looks good to me. I will work on this.
> > >
> >
> > One more thing we can do is to identify whether the tuple belongs to
> > toast relation while decoding it. However, I think to do that we need
> > to have access to relcache at that time and that might add some
> > overhead as we need to do that for each tuple. Can we investigate
> > what it will take to do that and if it is better than setting a bit
> > during WAL logging.
>
> IMHO, for the catalog scan, we will have to start/stop the transaction
> for each change. So do you want that we should evaluate its
> performance?
>
No, I was not thinking about each change, but at the level of ReorderBufferTXN.
> Also, during we get the change we might not have the
> complete historic snapshot ready to fetch the rel cache entry.
>
Before decoding each change (say DecodeInsert), we call
SnapBuildProcessChange. Isn't that sufficient?
Even, if the above is possible, I am not sure how good is it for each
change we fetch rel cache entry, that is the point I was worried.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 06:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 08:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-28 08:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 09:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-28 08:25 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 1:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:58 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > > > It seems to me that we need to add all of this new handling because
> > > > > while taking the decision whether to stream or not we don't know
> > > > > whether the txn has changes that can't be streamed. One idea to make
> > > > > it work is that we identify it while decoding the WAL. I think we
> > > > > need to set a bit in the insert/delete WAL record to identify if the
> > > > > tuple belongs to a toast relation. This won't add any additional
> > > > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > > > and also decoding will be efficient. If this is feasible, then we can
> > > > > do the same for speculative insertions.
> > > > The Idea looks good to me. I will work on this.
> > > >
> > >
> > > One more thing we can do is to identify whether the tuple belongs to
> > > toast relation while decoding it. However, I think to do that we need
> > > to have access to relcache at that time and that might add some
> > > overhead as we need to do that for each tuple. Can we investigate
> > > what it will take to do that and if it is better than setting a bit
> > > during WAL logging.
> >
> > IMHO, for the catalog scan, we will have to start/stop the transaction
> > for each change. So do you want that we should evaluate its
> > performance?
> >
>
> No, I was not thinking about each change, but at the level of ReorderBufferTXN.
That means we will have to keep that transaction open until we decode
the commit WAL for that ReorderBufferTXN or you have anything else in
mind?
>
> > Also, during we get the change we might not have the
> > complete historic snapshot ready to fetch the rel cache entry.
> >
>
> Before decoding each change (say DecodeInsert), we call
> SnapBuildProcessChange. Isn't that sufficient?
Yeah, Right, we can get some recache entry based on the base snapshot.
And, that might be sufficient to know whether it's a toast relation or
not.
>
> Even, if the above is possible, I am not sure how good is it for each
> change we fetch rel cache entry, that is the point I was worried.
We might not need to scan the catalog every time, we might get it from
the cache itself.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 06:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 08:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 08:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-28 09:06 ` Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-28 09:06 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 1:55 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 1:30 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2020 at 11:58 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > > > It seems to me that we need to add all of this new handling because
> > > > > > while taking the decision whether to stream or not we don't know
> > > > > > whether the txn has changes that can't be streamed. One idea to make
> > > > > > it work is that we identify it while decoding the WAL. I think we
> > > > > > need to set a bit in the insert/delete WAL record to identify if the
> > > > > > tuple belongs to a toast relation. This won't add any additional
> > > > > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > > > > and also decoding will be efficient. If this is feasible, then we can
> > > > > > do the same for speculative insertions.
> > > > > The Idea looks good to me. I will work on this.
> > > > >
> > > >
> > > > One more thing we can do is to identify whether the tuple belongs to
> > > > toast relation while decoding it. However, I think to do that we need
> > > > to have access to relcache at that time and that might add some
> > > > overhead as we need to do that for each tuple. Can we investigate
> > > > what it will take to do that and if it is better than setting a bit
> > > > during WAL logging.
> > >
> > > IMHO, for the catalog scan, we will have to start/stop the transaction
> > > for each change. So do you want that we should evaluate its
> > > performance?
> > >
> >
> > No, I was not thinking about each change, but at the level of ReorderBufferTXN.
> That means we will have to keep that transaction open until we decode
> the commit WAL for that ReorderBufferTXN or you have anything else in
> mind?
>
or probably till we start streaming.
> >
> > > Also, during we get the change we might not have the
> > > complete historic snapshot ready to fetch the rel cache entry.
> > >
> >
> > Before decoding each change (say DecodeInsert), we call
> > SnapBuildProcessChange. Isn't that sufficient?
> Yeah, Right, we can get some recache entry based on the base snapshot.
> And, that might be sufficient to know whether it's a toast relation or
> not.
> >
> > Even, if the above is possible, I am not sure how good is it for each
> > change we fetch rel cache entry, that is the point I was worried.
>
> We might not need to scan the catalog every time, we might get it from
> the cache itself.
>
Right, but I am not completely sure if that is better than setting a
bit in WAL record for toast tuples.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 06:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-02-04 05:29 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 03:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-02-04 05:29 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:34 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2020 at 11:28 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Jan 22, 2020 at 10:30 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > On Tue, Jan 14, 2020 at 10:44 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Hmm, I think this can turn out to be inefficient because we can easily
> > > > > end up spilling the data even when we don't need to so. Consider
> > > > > cases, where part of the streamed changes are for toast, and remaining
> > > > > are the changes which we would have streamed and hence can be removed.
> > > > > In such cases, we could have easily consumed remaining changes for
> > > > > toast without spilling. Also, I am not sure if spilling changes from
> > > > > the hash table is a good idea as they are no more in the same order as
> > > > > they were in ReorderBuffer which means the order in which we serialize
> > > > > the changes normally would change and that might have some impact, so
> > > > > we would need some more study if we want to pursue this idea.
> > > > I have fixed this bug and attached it as a separate patch. I will
> > > > merge it to the main patch after we agree with the idea and after some
> > > > more testing.
> > > >
> > > > The idea is that whenever we get the toasted chunk instead of directly
> > > > inserting it into the toast hash I am inserting it into some local
> > > > list so that if we don't get the change for the main table then we can
> > > > insert these changes back to the txn->changes. So once we get the
> > > > change for the main table at that time I am preparing the hash table
> > > > to merge the chunks.
> > > >
> > >
> > >
> > > I think this idea will work but appears to be quite costly because (a)
> > > you might need to serialize/deserialize the changes multiple times and
> > > might attempt streaming multiple times even though you can't do (b)
> > > you need to remove/add the same set of changes from the main list
> > > multiple times.
> > I agree with this.
> > >
> > > It seems to me that we need to add all of this new handling because
> > > while taking the decision whether to stream or not we don't know
> > > whether the txn has changes that can't be streamed. One idea to make
> > > it work is that we identify it while decoding the WAL. I think we
> > > need to set a bit in the insert/delete WAL record to identify if the
> > > tuple belongs to a toast relation. This won't add any additional
> > > overhead in WAL and reduce a lot of complexity in the logical decoding
> > > and also decoding will be efficient. If this is feasible, then we can
> > > do the same for speculative insertions.
> > The Idea looks good to me. I will work on this.
> >
>
> One more thing we can do is to identify whether the tuple belongs to
> toast relation while decoding it. However, I think to do that we need
> to have access to relcache at that time and that might add some
> overhead as we need to do that for each tuple. Can we investigate
> what it will take to do that and if it is better than setting a bit
> during WAL logging.
>
I have done some more analysis on this and it appears that there are
few problems in doing this. Basically, once we get the confirmed
flush location, we advance the replication_slot_catalog_xmin so that
vacuum can garbage collect the old tuple. So the problem is that
while we are collecting the changes in the ReorderBuffer our catalog
version might have removed, and we might not find any relation entry
with that relfilenodeid (because it is dropped or altered in the
future).
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 06:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-04 05:29 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-02-05 03:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:18 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-02-05 03:57 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Feb 4, 2020 at 11:00 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > One more thing we can do is to identify whether the tuple belongs to
> > toast relation while decoding it. However, I think to do that we need
> > to have access to relcache at that time and that might add some
> > overhead as we need to do that for each tuple. Can we investigate
> > what it will take to do that and if it is better than setting a bit
> > during WAL logging.
> >
> I have done some more analysis on this and it appears that there are
> few problems in doing this. Basically, once we get the confirmed
> flush location, we advance the replication_slot_catalog_xmin so that
> vacuum can garbage collect the old tuple. So the problem is that
> while we are collecting the changes in the ReorderBuffer our catalog
> version might have removed, and we might not find any relation entry
> with that relfilenodeid (because it is dropped or altered in the
> future).
>
Hmm, this means this can also occur while streaming the changes. The
main reason as I understand is that it is because before decoding
commit, we don't know whether these changes are already sent to the
subscriber (based on confirmed_flush_location/start_decoding_at). I
think it is better to skip streaming such transactions as we can't
make the right decision about these and as this can happen generally
after the crash for the first few transactions, it shouldn't matter
much if we serialize such transactions instead of streaming them.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-13 09:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 06:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-04 05:29 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 03:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-02-05 04:18 ` Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-02-05 04:18 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Feb 5, 2020 at 9:27 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Feb 4, 2020 at 11:00 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Jan 28, 2020 at 11:43 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > One more thing we can do is to identify whether the tuple belongs to
> > > toast relation while decoding it. However, I think to do that we need
> > > to have access to relcache at that time and that might add some
> > > overhead as we need to do that for each tuple. Can we investigate
> > > what it will take to do that and if it is better than setting a bit
> > > during WAL logging.
> > >
> > I have done some more analysis on this and it appears that there are
> > few problems in doing this. Basically, once we get the confirmed
> > flush location, we advance the replication_slot_catalog_xmin so that
> > vacuum can garbage collect the old tuple. So the problem is that
> > while we are collecting the changes in the ReorderBuffer our catalog
> > version might have removed, and we might not find any relation entry
> > with that relfilenodeid (because it is dropped or altered in the
> > future).
> >
>
> Hmm, this means this can also occur while streaming the changes. The
> main reason as I understand is that it is because before decoding
> commit, we don't know whether these changes are already sent to the
> subscriber (based on confirmed_flush_location/start_decoding_at).
Right.
>I think it is better to skip streaming such transactions as we can't
> make the right decision about these and as this can happen generally
> after the crash for the first few transactions, it shouldn't matter
> much if we serialize such transactions instead of streaming them.
I think the idea makes sense to me.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-10 04:44 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 21:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-01-10 04:44 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 9:21 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Sat, Jan 4, 2020 at 4:07 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > It is better to merge it with the main patch for
> > > "Implement-streaming-mode-in-ReorderBuffer", otherwise, it is a bit
> > > difficult to review.
> > Actually, we can merge 0008, 0009, 0012, 0018 to the main patch
> > (0007). Basically, if we merge all of them then we don't need to deal
> > with the conflict. I think Tomas has kept them separate so that we
> > can review the solution for the schema sent. And, I kept 0018 as a
> > separate patch to avoid conflict and rebasing in 0008, 0009 and 0012.
> > In the next patch set, I will merge all of them to 0007.
> >
>
> Okay, I think we can merge those patches.
Done
0008, 0009, 0017, 0018 are merged to 0007, 0012 is merged to 0010
>
> Few more comments:
> --------------------------------
> v4-0007-Implement-streaming-mode-in-ReorderBuffer
> 1.
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> + /*
> + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> + * information about
> subtransactions, which could arrive after streaming start.
> + */
> + if (!txn->is_schema_sent)
> + snapshot_now
> = ReorderBufferCopySnap(rb, txn->base_snapshot,
> + txn,
> command_id);
> ..
> }
>
> Why are we using base snapshot here instead of the snapshot we saved
> the first time streaming has happened? And as mentioned in comments,
> won't we need to consider the snapshots for subtransactions that
> arrived after the last time we have streamed the changes?
Fixed
>
> 2.
> + /* remember the command ID and snapshot for the streaming run */
> + txn->command_id = command_id;
> + txn-
> >snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
> +
> txn, command_id);
>
> I don't see where the txn->snapshot_now is getting freed. The
> base_snapshot is freed in ReorderBufferCleanupTXN, but I don't see
> this getting freed.
I have freed this In ReorderBufferCleanupTXN
>
> 3.
> +static void
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> + /*
> + * If this is a subxact, we need to stream the top-level transaction
> + * instead.
> + */
> + if (txn->toptxn)
> + {
> +
> ReorderBufferStreamTXN(rb, txn->toptxn);
> + return;
> + }
>
> Is it ever possible that we reach here for subtransaction, if not,
> then it should be Assert rather than if condition?
Fixed
>
> 4. In ReorderBufferStreamTXN(), don't we need to set some of the txn
> fields like origin_id, origin_lsn as we do in ReorderBufferCommit()
> especially to cover the case when it gets called due to memory
> overflow (aka via ReorderBufferCheckMemoryLimit).
We get origin_lsn during commit time so I am not sure how can we do
that. I have also noticed that currently, we are not using origin_lsn
on the subscriber side. I think need more investigation that if we
want this then do we need to log it early.
>
> v4-0017-Extend-handling-of-concurrent-aborts-for-streamin
> 1.
> @@ -3712,7 +3727,22 @@ ReorderBufferStreamTXN(ReorderBuffer *rb,
> ReorderBufferTXN *txn)
> if (using_subtxn)
>
> RollbackAndReleaseCurrentSubTransaction();
>
> - PG_RE_THROW();
> + /* re-throw only if it's not an abort */
> + if (errdata-
> >sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
> + {
> + MemoryContextSwitchTo(ecxt);
> + PG_RE_THROW();
> +
> }
> + else
> + {
> + /* remember the command ID and snapshot for the streaming run */
> + txn-
> >command_id = command_id;
> + txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
> +
> txn, command_id);
> + rb->stream_stop(rb, txn);
> +
> +
> FlushErrorState();
> + }
>
> Can you update comments either in the above code block or some other
> place to explain what is the concurrent abort problem and how we dealt
> with it? Also, please explain how the above error handling is
> sufficient to address all the various scenarios (sub-transaction got
> aborted when we have already sent some changes, or when we have not
> sent any changes yet).
Done
>
> v4-0006-Gracefully-handle-concurrent-aborts-of-uncommitte
> 1.
> + /*
> + * If CheckXidAlive is valid, then we check if it aborted. If it did, we
> + * error out
> + */
> + if (TransactionIdIsValid(CheckXidAlive) &&
> + !TransactionIdIsInProgress(CheckXidAlive) &&
> + !TransactionIdDidCommit(CheckXidAlive))
> + ereport(ERROR,
> + (errcode(ERRCODE_TRANSACTION_ROLLBACK),
> + errmsg("transaction aborted during system catalog scan")));
>
> Why here we can't use TransactionIdDidAbort? If we can't use it, then
> can you add comments stating the reason of the same.
Done
>
> 2.
> /*
> + * An xid value pointing to a possibly ongoing or a prepared transaction.
> + * Currently used in logical decoding. It's possible that such transactions
> + * can get aborted while the decoding is ongoing.
> + */
> +TransactionId CheckXidAlive = InvalidTransactionId;
>
> In comments, there is a mention of a prepared transaction. Do we
> allow prepared transactions to be decoded as part of this patch?
Fixed
>
> 3.
> + /*
> + * If CheckXidAlive is valid, then we check if it aborted. If it did, we
> + * error out
> + */
> + if (TransactionIdIsValid
> (CheckXidAlive) &&
> + !TransactionIdIsInProgress(CheckXidAlive) &&
> + !TransactionIdDidCommit(CheckXidAlive))
>
> This comment just says what code below is doing, can you explain the
> rationale behind this check. It would be better if it is clear by
> reading comments, why we are doing this check after fetching the
> tuple. I think this can refer to the comment I suggested to add for
> changes in patch
> v4-0017-Extend-handling-of-concurrent-aborts-for-streamin.
Done
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v5-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.3K) download
From 9924675dc35109c8823ee5a748e293d6917de1b7 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v5 01/14] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 017f03b..51557e2 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 2fa0a7f..b11b0c2 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 3aa6812..8c281e8 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 5e1dc8a..a99fcaf 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 98b033f..e23892a 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index f7cc8c4..4805cae 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v5-0004-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From 2c2dbdf2ea938c8115a88c83255a77ecac371bec Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v5 04/14] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index cd105d9..a3efbfd 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..ace21ec 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index bdf4389..9c95fc1 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index d16bebf..9e84687 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -345,6 +345,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -384,6 +430,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v5-0002-Issue-individual-invalidations-with-wal_level-log.patch (application/octet-stream, 16.4K) download
From d35181a5128caaa300131f5a9674d15614cc0f57 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v5 02/14] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 50 +++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 ++++++++
src/backend/replication/logical/reorderbuffer.c | 56 +++++++++++++++---
src/backend/utils/cache/inval.c | 75 +++++++++++++++++++++++++
src/include/access/xact.h | 18 +++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 235 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index e388cc7..6e46d19 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,44 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 51557e2..dd3d36f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a99fcaf..13a11ac 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f352805..723300b 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -473,6 +473,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1822,17 +1823,23 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ txn->is_schema_sent = false;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2227,6 +2234,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2674,6 +2713,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2770,6 +2810,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3055,6 +3096,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..e0d04b9 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write individual invalidations into WAL to support
+ * the decoding of the in-progress transaction. As of now it was enough to
+ * log invalidation only at commit because we are only decoding the transaction
+ * at the commit time. We only need to log the catalog cache and relcache
+ * invalidation. There can not be any active MVCC scan in logical decoding so
+ * we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +499,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +523,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +545,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +565,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1552,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..6f2a583 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e42d4c5..d16bebf 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -448,6 +459,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v5-0005-Cleaning-up-of-flags-in-ReorderBufferTXN-structur.patch (application/octet-stream, 8.1K) download
From 3c783cce62294b92c51d9f4dba316443f25e7cee Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 18:08:37 +0200
Subject: [PATCH v5 05/14] Cleaning up of flags in ReorderBufferTXN structure
---
src/backend/replication/logical/reorderbuffer.c | 36 ++++++++++++-------------
src/include/replication/reorderbuffer.h | 33 ++++++++++++++---------
2 files changed, 38 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 652a76e..9eda992 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -741,7 +741,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_first_lsn < cur_txn->first_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_first_lsn = cur_txn->first_lsn;
}
@@ -761,7 +761,7 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
Assert(prev_base_snap_lsn < cur_txn->base_snapshot_lsn);
/* known-as-subtxn txns must not be listed */
- Assert(!cur_txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(cur_txn));
prev_base_snap_lsn = cur_txn->base_snapshot_lsn;
}
@@ -784,7 +784,7 @@ ReorderBufferGetOldestTXN(ReorderBuffer *rb)
txn = dlist_head_element(ReorderBufferTXN, node, &rb->toplevel_by_lsn);
- Assert(!txn->is_known_as_subxact);
+ Assert(!rbtxn_is_known_subxact(txn));
Assert(txn->first_lsn != InvalidXLogRecPtr);
return txn;
}
@@ -844,7 +844,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
if (!new_sub)
{
- if (subtxn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(subtxn))
{
/* already associated, nothing to do */
return;
@@ -860,7 +860,7 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
}
}
- subtxn->is_known_as_subxact = true;
+ subtxn->txn_flags |= RBTXN_IS_SUBXACT;
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
@@ -1081,7 +1081,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
{
ReorderBufferChange *cur_change;
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1110,7 +1110,7 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
{
ReorderBufferChange *cur_change;
- if (cur_txn->serialized)
+ if (rbtxn_is_serialized(cur_txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, cur_txn);
@@ -1274,7 +1274,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
* they originally were happening inside another subtxn, so we won't
* ever recurse more than one level deep here.
*/
- Assert(subtxn->is_known_as_subxact);
+ Assert(rbtxn_is_known_subxact(subtxn));
Assert(subtxn->nsubtxns == 0);
ReorderBufferCleanupTXN(rb, subtxn);
@@ -1322,7 +1322,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/*
* Remove TXN from its containing list.
*
- * Note: if txn->is_known_as_subxact, we are deleting the TXN from its
+ * Note: if txn is known as subxact, we are deleting the TXN from its
* parent's list of known subxacts; this leaves the parent's nsubxacts
* count too high, but we don't care. Otherwise, we are deleting the TXN
* from the LSN-ordered list of toplevel TXNs.
@@ -1337,7 +1337,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
Assert(found);
/* remove entries spilled to disk */
- if (txn->serialized)
+ if (rbtxn_is_serialized(txn))
ReorderBufferRestoreCleanup(rb, txn);
/* deallocate */
@@ -1354,7 +1354,7 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!txn->has_catalog_changes || dlist_is_empty(&txn->tuplecids))
+ if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
return;
memset(&hash_ctl, 0, sizeof(hash_ctl));
@@ -1987,7 +1987,7 @@ ReorderBufferAbortOld(ReorderBuffer *rb, TransactionId oldestRunningXid)
* final_lsn to that of their last change; this causes
* ReorderBufferRestoreCleanup to do the right thing.
*/
- if (txn->serialized && txn->final_lsn == 0)
+ if (rbtxn_is_serialized(txn) && txn->final_lsn == 0)
{
ReorderBufferChange *last =
dlist_tail_element(ReorderBufferChange, node, &txn->changes);
@@ -2135,7 +2135,7 @@ ReorderBufferSetBaseSnapshot(ReorderBuffer *rb, TransactionId xid,
* operate on its top-level transaction instead.
*/
txn = ReorderBufferTXNByXid(rb, xid, true, &is_new, lsn, true);
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
Assert(txn->base_snapshot == NULL);
@@ -2314,7 +2314,7 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- txn->has_catalog_changes = true;
+ txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2331,7 +2331,7 @@ ReorderBufferXidHasCatalogChanges(ReorderBuffer *rb, TransactionId xid)
if (txn == NULL)
return false;
- return txn->has_catalog_changes;
+ return rbtxn_has_catalog_changes(txn);
}
/*
@@ -2351,7 +2351,7 @@ ReorderBufferXidHasBaseSnapshot(ReorderBuffer *rb, TransactionId xid)
return false;
/* a known subtxn? operate on top-level txn instead */
- if (txn->is_known_as_subxact)
+ if (rbtxn_is_known_subxact(txn))
txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, false,
NULL, InvalidXLogRecPtr, false);
@@ -2539,12 +2539,12 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
rb->spillBytes += size;
/* Don't consider already serialized transaction. */
- rb->spillTxns += txn->serialized ? 0 : 1;
+ rb->spillTxns += rbtxn_is_serialized(txn) ? 0 : 1;
Assert(spilled == txn->nentries_mem);
Assert(dlist_is_empty(&txn->changes));
txn->nentries_mem = 0;
- txn->serialized = true;
+ txn->txn_flags |= RBTXN_IS_SERIALIZED;
if (fd != -1)
CloseTransientFile(fd);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 9e84687..04dd0cb 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -169,18 +169,34 @@ typedef struct ReorderBufferChange
dlist_node node;
} ReorderBufferChange;
+/* ReorderBufferTXN flags */
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+
+/* does the txn have catalog changes */
+#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
+/* is the txn known as a subxact? */
+#define rbtxn_is_known_subxact(txn) (txn->txn_flags & RBTXN_IS_SUBXACT)
+/*
+ * Has this transaction been spilled to disk? It's not always possible to
+ * deduce that fact by comparing nentries with nentries_mem, because e.g.
+ * subtransactions of a large transaction might get serialized together
+ * with the parent - if they're restored to memory they'd have
+ * nentries_mem == nentries.
+ */
+#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+
typedef struct ReorderBufferTXN
{
+ int txn_flags;
+
/*
* The transactions transaction id, can be a toplevel or sub xid.
*/
TransactionId xid;
- /* did the TX have catalog changes */
- bool has_catalog_changes;
-
/* Do we know this is a subxact? Xid of top-level txn if so */
- bool is_known_as_subxact;
TransactionId toplevel_xid;
/*
@@ -249,15 +265,6 @@ typedef struct ReorderBufferTXN
uint64 nentries_mem;
/*
- * Has this transaction been spilled to disk? It's not always possible to
- * deduce that fact by comparing nentries with nentries_mem, because e.g.
- * subtransactions of a large transaction might get serialized together
- * with the parent - if they're restored to memory they'd have
- * nentries_mem == nentries.
- */
- bool serialized;
-
- /*
* List of ReorderBufferChange structs, including new Snapshots and new
* CommandIds
*/
--
1.8.3.1
v5-0003-fixup-is_schema_sent-set-too-early.patch (application/octet-stream, 858B) download
From bd1088604c7917441322bcdac6af6c45c4cf765c Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 22:50:55 +0100
Subject: [PATCH v5 03/14] fixup: is_schema_sent set too early
---
src/backend/replication/logical/reorderbuffer.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 723300b..652a76e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1837,7 +1837,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* about the message itself?
*/
LocalExecuteInvalidationMessage(&change->data.inval.msg);
- txn->is_schema_sent = false;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
--
1.8.3.1
v5-0006-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (application/octet-stream, 13.1K) download
From 74663de44159dfe79a6d6dca3f1d8b7d0453a886 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v5 06/14] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++
src/backend/access/index/genam.c | 49 +++++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++--
src/backend/utils/time/snapmgr.c | 25 ++++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 124 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index ace21ec..319349a 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 7b8490d..2d4ef48 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1431,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1552,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1708,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5514,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index c16eb05..5644b8d 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,22 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +530,22 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +672,22 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 9eda992..c9d28a3 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -692,7 +692,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1551,7 +1551,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1802,7 +1802,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +1822,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v5-0007-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 37.2K) download
From 60d03b120ef3ca7a62be67d78294322fb9dd5341 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 6 Jan 2020 13:15:24 +0530
Subject: [PATCH v5 07/14] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 691 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 32 ++
3 files changed, 668 insertions(+), 93 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c9d28a3..c381d8d 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -769,6 +782,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -864,6 +909,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -987,7 +1035,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1023,6 +1071,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1037,6 +1088,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1320,6 +1374,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1345,8 +1408,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1354,9 +1502,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1495,63 +1640,48 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * build data to be able to lookup the CommandIds of catalog tuples
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
-
- snapshot_now = txn->base_snapshot;
-
- /* build data to be able to lookup the CommandIds of catalog tuples */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1567,15 +1697,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1583,6 +1718,35 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ if (streaming)
+ {
+ /*
+ * While streaming an in-progress transaction there is a
+ * possibility that the (sub)transaction might get aborted
+ * concurrently. In such case if the (sub)transaction has
+ * catalog update then we might decode the tuple using wrong
+ * catalog version. So for detecting the concurrent abort we
+ * set CheckXidAlive to the current (sub)transaction's xid for
+ * which this change belongs to. And, during catalog scan we
+ * can check the status of the xid and if it is aborted we will
+ * report an specific error which we can ignore. We might have
+ * already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the
+ * abort we will stream abort message to truncate the changes in
+ * the subscriber.
+ */
+ CheckXidAlive = change->txn->xid;
+ }
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1592,8 +1756,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1659,7 +1821,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1680,8 +1850,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1699,7 +1867,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1757,7 +1925,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1766,10 +1942,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1800,9 +1982,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +2004,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1860,14 +2043,40 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1885,14 +2094,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then Discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,18 +2128,117 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1946,6 +2262,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2030,6 +2353,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2165,8 +2495,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2174,6 +2513,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2185,19 +2525,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2226,6 +2575,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2315,6 +2665,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2419,6 +2776,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2438,15 +2827,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2739,6 +3159,101 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after we there
+ * might be some new sub-transaction which after the last streaming run
+ * so we need to add those sub-xip in the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 04dd0cb..c4a2643 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* does the txn have catalog changes */
#define rbtxn_has_catalog_changes(txn) (txn->txn_flags & RBTXN_HAS_CATALOG_CHANGES)
@@ -187,6 +188,20 @@ typedef struct ReorderBufferChange
*/
#define rbtxn_is_serialized(txn) (txn->txn_flags & RBTXN_IS_SERIALIZED)
+/*
+ * Has this transaction been streamed to downstream? Similarly to spilling
+ * to disk, it's not trivial to deduce this from nentries and nentries_mem,
+ * for various reasons. For example, all changes may be in subtransactions
+ * in which case we'd have nentries==0 for the toplevel one, and it'd say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.
+ *
+ * Note: We never stream and serialize a transaction at the same time (e
+ * only do spill to disk when streaming is not supported by the plugin),
+ * so only one of those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) (txn->txn_flags & RBTXN_IS_STREAMED)
+
typedef struct ReorderBufferTXN
{
int txn_flags;
@@ -222,6 +237,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -252,6 +277,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v5-0009-Support-logical_decoding_work_mem-set-from-create.patch (application/octet-stream, 13.1K) download
From 70b890b5e730d7c08661c3bfd6eecf149d766f2d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v5 09/14] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5d1c902..8b1923c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index f77a83b..5cd1daa 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 95962b4..c45c2ce 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 42e3e04..16f9d00 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7a5471f..48b960c 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1745,6 +1745,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 7525082..536722b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..3394379 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index a276237..66e89f0 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
v5-0008-Fix-speculative-insert-bug.patch (application/octet-stream, 2.5K) download
From 490495a4cd8ac57753f9977122bbd9887f4c09c1 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 10 Jan 2020 09:01:35 +0530
Subject: [PATCH v5 08/14] Fix speculative insert bug.
---
src/backend/replication/logical/reorderbuffer.c | 23 +++++++++++++++++++----
src/include/replication/reorderbuffer.h | 6 ++++++
2 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c381d8d..eb6fda5 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1701,6 +1701,16 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
+ /*
+ * Resotre any previous speculative inserted tuple if we are running in
+ * streaming mode.
+ */
+ if (streaming && txn->specinsert != NULL)
+ {
+ specinsert = txn->specinsert;
+ txn->specinsert = NULL;
+ }
+
if (using_subtxn)
BeginInternalSubTransaction("stream");
else
@@ -2029,13 +2039,18 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
- * There's a speculative insertion remaining, just clean in up, it
- * can't have been successful, otherwise we'd gotten a confirmation
- * record.
+ * In non-streaming mode if there's a speculative insertion remaining,
+ * just clean in up, it can't have been successful, otherwise we'd
+ * gotten a confirmation record. For streaming mode, remember the tuple
+ * so that if we get the confirmation in the next stream we can stream
+ * it.
*/
if (specinsert)
{
- ReorderBufferReturnChange(rb, specinsert);
+ if (streaming)
+ txn->specinsert = specinsert;
+ else
+ ReorderBufferReturnChange(rb, specinsert);
specinsert = NULL;
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index c4a2643..f41e216 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -335,6 +335,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Speculative insert saved from the last streamed run if the speculative
+ * confirm has not received in the same stream.
+ */
+ ReorderBufferChange *specinsert;
+
/* ---
* Position in one of three lists:
* * list of subtransactions if we are *known* to be subxact
--
1.8.3.1
v5-0010-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 91.3K) download
From 8af095c64b08a228fdfdae2e6fdee5d3b686b9f2 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:05:20 +0100
Subject: [PATCH v5 10/14] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 310 +++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2074 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 5cd1daa..1dc486c 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index c45c2ce..1ece10d 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 51c486b..03ef76c 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 16f9d00..61701d0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..e80d00c 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 9c95fc1..61064f3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index dcf7c08..918a841 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 48b960c..5c20c0e 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -529,6 +636,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found PG_USED_FOR_ASSERTS_ONLY = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -695,6 +1123,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -830,6 +1261,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -929,6 +1363,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1020,6 +1457,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1117,6 +1570,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1132,6 +1601,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1580,6 +2052,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1746,6 +2776,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 536722b..ebe0423 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,17 +45,45 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -64,11 +92,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -84,16 +118,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +206,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +235,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +259,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +280,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +368,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +435,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +463,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +495,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +512,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +521,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +547,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +571,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +652,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -549,6 +773,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*/
static RelationSyncEntry *
@@ -623,6 +875,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -657,7 +939,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index bb69683..3085c0f 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 9c06374..63fc2c7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -969,6 +969,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3394379..18f416f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e5a5d02..bbc9112 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2cc2dc4..ade4188 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 66e89f0..1e4269c 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v5-0011-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From 6f2811a82d859b692b6b1a8e59083c6ab5ac953b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Jan 2020 09:45:27 +0530
Subject: [PATCH v5 11/14] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dcb5811..180ea88 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1996,6 +1996,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 773edf8..cb9e6ee 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -785,7 +785,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index eb6fda5..9dd379e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3264,6 +3268,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 63fc2c7..d7f22ae 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1293,7 +1293,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1314,7 +1314,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2357,6 +2358,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3196,7 +3200,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3253,6 +3257,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3276,6 +3283,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3362,6 +3372,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3610,11 +3625,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 427faa3..9ef4fbf 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5173,9 +5173,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index f41e216..d671a2f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -517,15 +517,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f..3888b0c 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62eaf90..2dcb063 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1960,9 +1960,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v5-0013-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1013B) download
From 657b9d3dd5df36dc04aa8138b071bb8eb3165024 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v5 13/14] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 9dd379e..5e8a931 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1327,6 +1327,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
v5-0014-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From fd866d4e41a1fe3d61606029b529998eb5403c14 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v5 14/14] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v5-0012-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From c01738e8cc2480efbc13af94b050ff83607eb589 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v5 12/14] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 77a1560..8cd1993 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -65,7 +65,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-10 21:35 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-10 21:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Alvaro Herrera @ 2020-01-10 21:35 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
Here's a rebase of this patch series. I didn't change anything except
1. disregard what was 0005, since I already pushed it.
2. roll 0003 into 0002.
3. rebase 0007 (now 0005) to account for the reorderbuffer changes.
(I did notice that 0005 adds a new boolean any_data_sent, which is
silly -- it should be another txn_flags bit.)
However, tests don't pass for me; notably, test_decoding crashes.
OTOH I noticed that the streamed transaction support in test_decoding
writes the XID to the output, which is going to make it useless for
regression testing. It probably should not emit the numerical values.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 21:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2020-01-10 21:37 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-11 02:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-14 05:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Alvaro Herrera @ 2020-01-10 21:37 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-Jan-10, Alvaro Herrera wrote:
> Here's a rebase of this patch series. I didn't change anything except
... this time with attachments ...
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
v6-0001-Immediately-WAL-log-assignments.patch (text/x-diff, 10.4K) download
From f2cd3a14f9513b880becedd95c933dad53c7d9e3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v6 01/12] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++--
src/backend/access/transam/xlogreader.c | 5 +++
src/backend/replication/logical/decode.c | 39 ++++++++++----------
src/include/access/xact.h | 3 ++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 ++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 017f03b6d8..51557e2951 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 2fa0a7f667..b11b0c2940 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 3aa68127a3..8c281e821a 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 5e1dc8a651..a99fcaf0a1 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04babc2..8645b3816c 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 98b033fc20..e23892ab87 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index f7cc8c4e1d..4805caec67 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0194..2f0c8bf589 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
2.20.1
v6-0002-Issue-individual-invalidations-with-wal_level-log.patch (text/x-diff, 16.3K) download
From ccc7defbeec7653240c76e6e4dcdb9349b49284d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v6 02/12] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 50 +++++++++++++
src/backend/access/transam/xact.c | 7 ++
src/backend/replication/logical/decode.c | 23 ++++++
.../replication/logical/reorderbuffer.c | 55 ++++++++++++--
src/backend/utils/cache/inval.c | 75 +++++++++++++++++++
src/include/access/xact.h | 18 ++++-
src/include/replication/reorderbuffer.h | 14 ++++
7 files changed, 234 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index e388cc714a..6e46d19168 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,44 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 51557e2951..dd3d36ffb1 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a99fcaf0a1..13a11ac782 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index bbd908af05..531897cf05 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -473,6 +473,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1822,17 +1823,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2225,6 +2231,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
/*
* Setup the invalidation of the toplevel transaction.
*
@@ -2674,6 +2712,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2770,6 +2809,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3055,6 +3095,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33be6..e0d04b9850 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write individual invalidations into WAL to support
+ * the decoding of the in-progress transaction. As of now it was enough to
+ * log invalidation only at commit because we are only decoding the transaction
+ * at the commit time. We only need to log the catalog cache and relcache
+ * invalidation. There can not be any active MVCC scan in logical decoding so
+ * we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +499,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +523,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +545,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +565,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1552,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b3816c..6f2a5831ee 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -197,6 +197,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
+/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4f6c65d6f4..fa41115db9 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -458,6 +469,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
2.20.1
v6-0003-Extend-the-output-plugin-API-with-stream-methods.patch (text/x-diff, 34.8K) download
From 1236dc75d5609f14d359afb66a0a5f8bbfbf353e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v6 03/12] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 ++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 +++++
src/include/replication/reorderbuffer.h | 57 ++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index cd105d91e0..a3efbfdd76 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db968641e..ace21ec8e5 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index bdf4389a57..9c95fc1ed8 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f1da..f24e2468ac 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -79,6 +79,11 @@ typedef struct LogicalDecodingContext
*/
void *output_writer_private;
+ /*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
/*
* State for writing output.
*/
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236c57..0d0a94a648 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -99,6 +99,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
*/
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
+/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
/*
* Output plugin callbacks
*/
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index fa41115db9..79ea33cd26 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -355,6 +355,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -393,6 +439,17 @@ struct ReorderBuffer
ReorderBufferCommitCB commit;
ReorderBufferMessageCB message;
+ /*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
/*
* Pointer that will be passed untouched to the callbacks.
*/
--
2.20.1
v6-0004-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (text/x-diff, 13.0K) download
From 06973f7b57a9c186e53400e8815d8edf7a6bc047 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v6 04/12] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 +-
src/backend/access/heap/heapam.c | 41 ++++++++++++++++
src/backend/access/index/genam.c | 49 +++++++++++++++++++
.../replication/logical/reorderbuffer.c | 8 +--
src/backend/utils/time/snapmgr.c | 25 +++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 124 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index ace21ec8e5..319349a92d 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 7b8490d4e5..2d4ef48069 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1421,6 +1430,14 @@ heap_fetch(Relation relation,
OffsetNumber offnum;
bool valid;
+ /*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
/*
* Fetch and pin the appropriate page of the relation.
*/
@@ -1535,6 +1552,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1682,6 +1707,14 @@ heap_get_latest_tid(TableScanDesc sscan,
*/
Assert(ItemPointerIsValid(tid));
+ /*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
/*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
@@ -5481,6 +5514,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index c16eb05416..5644b8d41a 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,22 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +530,22 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +672,22 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 531897cf05..2da0a23a7e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -692,7 +692,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1551,7 +1551,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1802,7 +1802,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +1822,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c592c..93a0c048c5 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -153,6 +153,13 @@ static Snapshot SecondarySnapshot = NULL;
static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
+/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13ce84..12f737b21a 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
2.20.1
v6-0005-Implement-streaming-mode-in-ReorderBuffer.patch (text/x-diff, 37.0K) download
From 84b94c7c49ca7aae737ffb1451eee9098c483578 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v6 05/12] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
.../replication/logical/reorderbuffer.c | 691 +++++++++++++++---
src/include/replication/reorderbuffer.h | 36 +
3 files changed, 672 insertions(+), 93 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba10890aa..160b167adb 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 2da0a23a7e..50341a6d9e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -768,6 +781,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
#endif
}
+/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
@@ -864,6 +909,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -987,7 +1035,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1023,6 +1071,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1037,6 +1088,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1319,6 +1373,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_delete(&txn->base_snapshot_node);
}
+ /*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
/*
* Remove TXN from its containing list.
*
@@ -1344,9 +1407,94 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferReturnTXN(rb, txn);
}
+/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1354,9 +1502,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1495,63 +1640,48 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
+
+ ReorderBufferStreamTXN(rb, txn);
+
+ rb->stream_commit(rb, txn, txn->final_lsn);
+
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
ReorderBufferIterTXNState *volatile iterstate = NULL;
-
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
-
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
-
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * build data to be able to lookup the CommandIds of catalog tuples
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
-
- snapshot_now = txn->base_snapshot;
-
- /* build data to be able to lookup the CommandIds of catalog tuples */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1567,15 +1697,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1583,6 +1718,35 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ if (streaming)
+ {
+ /*
+ * While streaming an in-progress transaction there is a
+ * possibility that the (sub)transaction might get aborted
+ * concurrently. In such case if the (sub)transaction has
+ * catalog update then we might decode the tuple using wrong
+ * catalog version. So for detecting the concurrent abort we
+ * set CheckXidAlive to the current (sub)transaction's xid for
+ * which this change belongs to. And, during catalog scan we
+ * can check the status of the xid and if it is aborted we will
+ * report an specific error which we can ignore. We might have
+ * already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the
+ * abort we will stream abort message to truncate the changes in
+ * the subscriber.
+ */
+ CheckXidAlive = change->txn->xid;
+ }
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1592,8 +1756,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1659,7 +1821,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1680,8 +1850,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1699,7 +1867,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1757,7 +1925,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1766,10 +1942,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1800,9 +1982,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +2004,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1860,14 +2043,40 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1885,14 +2094,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then Discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,17 +2128,116 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
- PG_RE_THROW();
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
+/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
@@ -1946,6 +2262,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2030,6 +2353,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2165,8 +2495,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2174,6 +2513,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2185,19 +2525,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2226,6 +2575,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2315,6 +2665,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2418,6 +2775,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
return largest;
}
+/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
@@ -2438,15 +2827,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
- ReorderBufferSerializeTXN(rb, txn);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2739,6 +3159,101 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after we there
+ * might be some new sub-transaction which after the last streaming run
+ * so we need to add those sub-xip in the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 79ea33cd26..629eeca7f6 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -192,6 +193,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -225,6 +244,16 @@ typedef struct ReorderBufferTXN
*/
XLogRecPtr final_lsn;
+ /*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
/*
* LSN pointing to the end of the commit record + 1.
*/
@@ -255,6 +284,13 @@ typedef struct ReorderBufferTXN
XLogRecPtr base_snapshot_lsn;
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
+ /*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
/*
* How many ReorderBufferChange's do we have in this txn.
*
--
2.20.1
v6-0006-Fix-speculative-insert-bug.patch (text/x-diff, 2.5K) download
From ae7bfc6848143185d13adaa5532c13e9f3d730ca Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 10 Jan 2020 09:01:35 +0530
Subject: [PATCH v6 06/12] Fix speculative insert bug.
---
.../replication/logical/reorderbuffer.c | 23 +++++++++++++++----
src/include/replication/reorderbuffer.h | 6 +++++
2 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 50341a6d9e..8e4744f73a 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1701,6 +1701,16 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
+ /*
+ * Resotre any previous speculative inserted tuple if we are running in
+ * streaming mode.
+ */
+ if (streaming && txn->specinsert != NULL)
+ {
+ specinsert = txn->specinsert;
+ txn->specinsert = NULL;
+ }
+
if (using_subtxn)
BeginInternalSubTransaction("stream");
else
@@ -2029,13 +2039,18 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
- * There's a speculative insertion remaining, just clean in up, it
- * can't have been successful, otherwise we'd gotten a confirmation
- * record.
+ * In non-streaming mode if there's a speculative insertion remaining,
+ * just clean in up, it can't have been successful, otherwise we'd
+ * gotten a confirmation record. For streaming mode, remember the tuple
+ * so that if we get the confirmation in the next stream we can stream
+ * it.
*/
if (specinsert)
{
- ReorderBufferReturnChange(rb, specinsert);
+ if (streaming)
+ txn->specinsert = specinsert;
+ else
+ ReorderBufferReturnChange(rb, specinsert);
specinsert = NULL;
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 629eeca7f6..0510d3831f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -343,6 +343,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Speculative insert saved from the last streamed run if the speculative
+ * confirm has not received in the same stream.
+ */
+ ReorderBufferChange *specinsert;
+
/* ---
* Position in one of three lists:
* * list of subtransactions if we are *known* to be subxact
--
2.20.1
v6-0007-Support-logical_decoding_work_mem-set-from-create.patch (text/x-diff, 13.1K) download
From c00cf7ddb5f49d355c52b704479373ac6598134b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v6 07/12] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 +++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 ++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5d1c90282f..8b1923c9de 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c244fb..91790b0c95 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index f77a83bd2e..5cd1daa238 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 95962b4a3e..c45c2ce212 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -697,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 42e3e04e68..16f9d008fd 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7a5471f95c..48b960c4c9 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1745,6 +1745,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 752508213a..536722b32f 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d42d8..3394379f86 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index a276237477..66e89f087b 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
2.20.1
v6-0008-Add-support-for-streaming-to-built-in-replication.patch (text/x-diff, 91.3K) download
From 934dcdd48e4720cb558e6e7a1049f80b7e218344 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@2ndquadrant.com>
Date: Fri, 27 Dec 2019 23:05:20 +0100
Subject: [PATCH v6 08/12] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 +++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 310 ++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../t/012_stream_subxact_abort.pl | 82 ++
.../t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2074 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4d3e..e1fb9075e1 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0c95..d9abf5e64c 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 5cd1daa238..1dc486c0e7 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index c45c2ce212..1ece10d9f5 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -692,11 +726,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -728,6 +765,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -740,7 +784,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -778,7 +822,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -815,7 +860,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 51c486bebd..03ef76caea 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 16f9d008fd..61701d0590 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e987..e80d00c1c3 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 9c95fc1ed8..61064f392a 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index dcf7c08c18..918a841125 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,7 +139,8 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
@@ -147,6 +148,10 @@ logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -182,8 +187,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -191,6 +196,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -252,13 +261,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -300,6 +314,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -309,6 +324,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -351,12 +370,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -401,7 +424,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -409,6 +432,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -689,3 +716,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 48b960c4c9..5c20c0e4c3 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -528,6 +635,318 @@ apply_handle_origin(StringInfo s)
errmsg("ORIGIN message sent out of order")));
}
+/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found PG_USED_FOR_ASSERTS_ONLY = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
/*
* Handle RELATION message.
*
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -695,6 +1123,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -830,6 +1261,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -929,6 +1363,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1020,6 +1457,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1116,6 +1569,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
}
+/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
/*
* Apply main loop.
*/
@@ -1132,6 +1601,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1580,6 +2052,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1746,6 +2776,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 536722b32f..ebe0423cc2 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,17 +45,45 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -64,11 +92,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -84,16 +118,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +206,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +235,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +259,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +280,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +368,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +435,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +463,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +495,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +512,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +521,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +547,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +571,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -512,6 +651,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
rel_sync_cache_publication_cb(arg, cacheid, hashvalue);
}
+/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
/*
* Initialize the relation schema sync cache for a decoding session.
*
@@ -548,6 +772,34 @@ init_rel_sync_cache(MemoryContext cachectx)
(Datum) 0);
}
+/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
/*
* Find or create entry in the relation schema cache.
*/
@@ -622,6 +874,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
return entry;
}
+/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
/*
* Relcache invalidation callback
*/
@@ -657,7 +939,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index bb69683e2a..3085c0f921 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 9c063749b6..63fc2c7ff2 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -968,6 +968,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndPrepareWrite, WalSndWriteData,
WalSndUpdateProgress);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
/*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3394379f86..18f416fa78 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e5a5d025ba..bbc9112b01 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2cc2dc4db3..ade4188dd2 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 66e89f087b..1e4269ca21 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000000..2f01133f69
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000000..d2ae38592b
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000000..0da39a1a8a
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000000..402df30f59
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000000..becbdd0578
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.20.1
v6-0009-Track-statistics-for-streaming.patch (text/x-diff, 11.7K) download
From 31d5e1ed8ced85ea01de18fbd70af0ad38956e55 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Jan 2020 09:45:27 +0530
Subject: [PATCH v6 09/12] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++
src/backend/catalog/system_views.sql | 5 ++-
.../replication/logical/reorderbuffer.c | 13 ++++++++
src/backend/replication/walsender.c | 32 ++++++++++++++++---
src/include/catalog/pg_proc.dat | 6 ++--
src/include/replication/reorderbuffer.h | 13 +++++---
src/include/replication/walsender_private.h | 5 +++
src/test/regress/expected/rules.out | 7 ++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dcb58115af..180ea880a4 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1996,6 +1996,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 773edf85e7..cb9e6ee9ea 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -785,7 +785,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 8e4744f73a..16515d0a12 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3264,6 +3268,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 63fc2c7ff2..d7f22ae960 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1293,7 +1293,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1314,7 +1314,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2357,6 +2358,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3196,7 +3200,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3253,6 +3257,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3276,6 +3283,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3362,6 +3372,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3610,11 +3625,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 427faa3c3b..9ef4fbf4f9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5173,9 +5173,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0510d3831f..7259c66c3f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -524,15 +524,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f0a4..3888b0c2f8 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62eaf90a0f..2dcb063912 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1960,9 +1960,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
2.20.1
v6-0010-Enable-streaming-for-all-subscription-TAP-tests.patch (text/x-diff, 14.8K) download
From 7d671806584fff71067c8bde38b2f642ba1331a9 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v6 10/12] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 77a1560b23..8cd1993393 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -65,7 +65,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2fbc..94c71f8ae2 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e7fe..ad3ed13ffc 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9181..a6fae9c3f1 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17a78..202871a658 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10a19..70c86b22ac 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6d63..f9c8d1d348 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7332..0c9c6b3dd4 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bdba9..21f50c7012 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133f69..30561d8f96 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae38592b..9a6bac6822 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bdc35..ed56fbf96c 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cba4c..4df1ddef63 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1a8a..c3caff6149 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7c2f..c62eb521e7 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30f59..2be7542831 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0578..2da9607a7d 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9435..96ffc091b0 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
2.20.1
v6-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (text/x-diff, 1016B) download
From ab09b5f50aa9af74da48148818b1899f907038eb Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v6 11/12] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 16515d0a12..c0c6a7f86c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1327,6 +1327,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
2.20.1
v6-0012-Add-TAP-test-for-streaming-vs.-DDL.patch (text/x-diff, 4.4K) download
From b9bb007d0dccb301acac5c0da13bd1d41e38428a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v6 12/12] Add TAP test for streaming vs. DDL
---
.../subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000000..b8d78b1972
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
2.20.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 21:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-10 21:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2020-01-11 02:48 ` Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 0 replies; 536+ messages in thread
From: Alvaro Herrera @ 2020-01-11 02:48 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-Jan-10, Alvaro Herrera wrote:
> From 7d671806584fff71067c8bde38b2f642ba1331a9 Mon Sep 17 00:00:00 2001
> From: Dilip Kumar <dilip.kumar@enterprisedb.com>
> Date: Wed, 20 Nov 2019 16:41:13 +0530
> Subject: [PATCH v6 10/12] Enable streaming for all subscription TAP tests
This patch turns a lot of test into the streamed mode. While it's
great that streaming mode is tested, we should add new tests for it
rather than failing to keep tests for the non-streamed mode. I suggest
that we add two versions of each test, one for each mode. Maybe the way
to do that is to create some subroutine that can be called twice.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 21:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-10 21:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2020-01-14 05:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-21 16:36 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-01-14 05:26 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Jan 11, 2020 at 3:07 AM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>
> On 2020-Jan-10, Alvaro Herrera wrote:
>
> > Here's a rebase of this patch series. I didn't change anything except
>
> ... this time with attachments ...
The patch set fails to apply on the head so rebased. (Rebased on
commit cebf9d6e6ee13cbf9f1a91ec633cf96780ffc985)
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v7-0002-Issue-individual-invalidations-with-wal_level-log.patch (application/octet-stream, 16.4K) download
From 6d920bb1d396c18fd0308a01d03ea105684bf388 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v7 02/12] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 50 +++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 ++++++++
src/backend/replication/logical/reorderbuffer.c | 55 +++++++++++++++---
src/backend/utils/cache/inval.c | 75 +++++++++++++++++++++++++
src/include/access/xact.h | 18 +++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 234 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index e388cc7..6e46d19 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -397,6 +402,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -424,7 +437,44 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 51557e2..dd3d36f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6001,6 +6001,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a99fcaf..13a11ac 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index bbd908a..531897c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -473,6 +473,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1822,17 +1823,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2227,6 +2233,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2674,6 +2712,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2770,6 +2809,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3055,6 +3095,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..e0d04b9 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write individual invalidations into WAL to support
+ * the decoding of the in-progress transaction. As of now it was enough to
+ * log invalidation only at commit because we are only decoding the transaction
+ * at the commit time. We only need to log the catalog cache and relcache
+ * invalidation. There can not be any active MVCC scan in logical decoding so
+ * we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +499,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +523,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +545,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +565,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1552,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..6f2a583 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4f6c65d..fa41115 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -458,6 +469,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v7-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.3K) download
From 6d97e11e93fc4de09af087c884014d54b21fb15e Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v7 01/12] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 017f03b..51557e2 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6002,3 +6004,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 2fa0a7f..b11b0c2 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 3aa6812..8c281e8 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1165,6 +1165,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1203,6 +1204,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 5e1dc8a..a99fcaf 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 98b033f..e23892a 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index f7cc8c4..4805cae 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -147,6 +147,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -280,6 +282,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v7-0005-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 37.1K) download
From e043a337a1b6bd390350e12792c8b632c52915f8 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v7 05/12] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 691 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 36 ++
3 files changed, 672 insertions(+), 93 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 2da0a23..50341a6 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -769,6 +782,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -864,6 +909,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -987,7 +1035,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1023,6 +1071,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1037,6 +1088,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1320,6 +1374,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1345,8 +1408,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1354,9 +1502,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1495,63 +1640,48 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * build data to be able to lookup the CommandIds of catalog tuples
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
-
- snapshot_now = txn->base_snapshot;
-
- /* build data to be able to lookup the CommandIds of catalog tuples */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1567,15 +1697,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1583,6 +1718,35 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ if (streaming)
+ {
+ /*
+ * While streaming an in-progress transaction there is a
+ * possibility that the (sub)transaction might get aborted
+ * concurrently. In such case if the (sub)transaction has
+ * catalog update then we might decode the tuple using wrong
+ * catalog version. So for detecting the concurrent abort we
+ * set CheckXidAlive to the current (sub)transaction's xid for
+ * which this change belongs to. And, during catalog scan we
+ * can check the status of the xid and if it is aborted we will
+ * report an specific error which we can ignore. We might have
+ * already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the
+ * abort we will stream abort message to truncate the changes in
+ * the subscriber.
+ */
+ CheckXidAlive = change->txn->xid;
+ }
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1592,8 +1756,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1659,7 +1821,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1680,8 +1850,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1699,7 +1867,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1757,7 +1925,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1766,10 +1942,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1800,9 +1982,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +2004,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1860,14 +2043,40 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1885,14 +2094,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then Discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,18 +2128,117 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1946,6 +2262,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2030,6 +2353,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2165,8 +2495,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2174,6 +2513,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2185,19 +2525,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2226,6 +2575,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2315,6 +2665,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2419,6 +2776,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2438,15 +2827,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2739,6 +3159,101 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after we there
+ * might be some new sub-transaction which after the last streaming run
+ * so we need to add those sub-xip in the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 79ea33c..629eeca 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -192,6 +193,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -226,6 +245,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -256,6 +285,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v7-0004-Gracefully-handle-concurrent-aborts-of-uncommitte.patch (application/octet-stream, 13.1K) download
From c6b0c427af35c8829a20f5ecc2f4ec6d93b3aaac Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v7 04/12] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++
src/backend/access/index/genam.c | 49 +++++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++--
src/backend/utils/time/snapmgr.c | 25 ++++++++++++-
src/include/utils/snapmgr.h | 4 +-
6 files changed, 124 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index ace21ec..319349a 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 7b8490d..2d4ef48 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1431,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1535,6 +1552,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1683,6 +1708,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5481,6 +5514,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index c16eb05..5644b8d 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,22 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +530,22 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +672,22 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 531897c..2da0a23 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -692,7 +692,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1551,7 +1551,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1802,7 +1802,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +1822,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v7-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From 7ef911c973808a9c6426a83d723040b6b862762d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v7 03/12] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index cd105d9..a3efbfd 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -542,3 +571,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..ace21ec 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index bdf4389..9c95fc1 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -863,6 +911,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index fa41115..79ea33c 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -355,6 +355,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -394,6 +440,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v7-0006-Fix-speculative-insert-bug.patch (application/octet-stream, 2.5K) download
From 719fb75d324938c2e397079776f437b6e00b765f Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 10 Jan 2020 09:01:35 +0530
Subject: [PATCH v7 06/12] Fix speculative insert bug.
---
src/backend/replication/logical/reorderbuffer.c | 23 +++++++++++++++++++----
src/include/replication/reorderbuffer.h | 6 ++++++
2 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 50341a6..8e4744f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1701,6 +1701,16 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
+ /*
+ * Resotre any previous speculative inserted tuple if we are running in
+ * streaming mode.
+ */
+ if (streaming && txn->specinsert != NULL)
+ {
+ specinsert = txn->specinsert;
+ txn->specinsert = NULL;
+ }
+
if (using_subtxn)
BeginInternalSubTransaction("stream");
else
@@ -2029,13 +2039,18 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
}
/*
- * There's a speculative insertion remaining, just clean in up, it
- * can't have been successful, otherwise we'd gotten a confirmation
- * record.
+ * In non-streaming mode if there's a speculative insertion remaining,
+ * just clean in up, it can't have been successful, otherwise we'd
+ * gotten a confirmation record. For streaming mode, remember the tuple
+ * so that if we get the confirmation in the next stream we can stream
+ * it.
*/
if (specinsert)
{
- ReorderBufferReturnChange(rb, specinsert);
+ if (streaming)
+ txn->specinsert = specinsert;
+ else
+ ReorderBufferReturnChange(rb, specinsert);
specinsert = NULL;
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 629eeca..0510d38 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -343,6 +343,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Speculative insert saved from the last streamed run if the speculative
+ * confirm has not received in the same stream.
+ */
+ ReorderBufferChange *specinsert;
+
/* ---
* Position in one of three lists:
* * list of subtransactions if we are *known* to be subxact
--
1.8.3.1
v7-0008-Add-support-for-streaming-to-built-in-replication.patch (application/octet-stream, 91.1K) download
From 38e572e314fd8f55e9359403568113bd642ee4ab Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 13 Jan 2020 14:24:39 +0530
Subject: [PATCH v7 08/12] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 5 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 60 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 8 +-
src/backend/replication/logical/launcher.c | 2 +
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 157 ++-
src/backend/replication/logical/worker.c | 1031 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 310 +++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2074 insertions(+), 42 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..e1fb907 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 91790b0..d9abf5e 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -218,6 +218,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index 5cd1daa..1dc486c 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -66,6 +66,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
sub->workmem = subform->subworkmem;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index c50e854..a486bd3 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -92,6 +93,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*refresh = true;
if (logical_wm)
*logical_wm_given = false;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -186,6 +189,26 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*logical_wm_given = true;
*logical_wm = defGetInt32(defel);
+
+ /*
+ * Check that the value is not smaller than 64kB (which is
+ * the minimum value for logical_work_mem).
+ */
+ if (*logical_wm < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("%d is outside the valid range for parameter \"work_mem\" (64 .. 2147483647)",
+ *logical_wm)));
+ }
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
}
else
ereport(ERROR,
@@ -332,6 +355,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool copy_data;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -348,7 +373,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL, &logical_wm, &logical_wm_given);
+ NULL, &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -430,7 +456,15 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subworkmem - 1] =
Int32GetDatum(logical_wm);
else
- nulls[Anum_pg_subscription_subworkmem - 1] = true;
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(-1);
+
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
@@ -691,11 +725,14 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *synchronous_commit;
int logical_wm;
bool logical_wm_given;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
NULL, &synchronous_commit, NULL,
- &logical_wm, &logical_wm_given);
+ &logical_wm, &logical_wm_given,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -727,6 +764,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subworkmem - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -739,7 +783,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
NULL, NULL, NULL, NULL, NULL,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -777,7 +821,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh, NULL, NULL);
+ NULL, &refresh, NULL, NULL,
+ NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -814,7 +859,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 51c486b..03ef76c 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 099d21b..470600a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,8 +406,12 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
- appendStringInfo(&cmd, ", work_mem '%d'",
- options->proto.logical.work_mem);
+ if (options->proto.logical.work_mem != -1)
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..e80d00c 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 9c95fc1..61064f3 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1149,7 +1149,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd..93780b2 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,10 +139,15 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -178,8 +183,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -187,6 +192,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -248,13 +257,18 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -296,6 +310,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -305,6 +320,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -347,12 +366,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -397,7 +420,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -405,6 +428,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -685,3 +712,119 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 48b960c..5c20c0e 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription. This
+ * is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -529,6 +636,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found PG_USED_FOR_ASSERTS_ONLY = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -695,6 +1123,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -830,6 +1261,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -929,6 +1363,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1020,6 +1457,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1117,6 +1570,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i]);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1132,6 +1601,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1580,6 +2052,564 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1746,6 +2776,7 @@ ApplyWorkerMain(Datum main_arg)
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
options.proto.logical.work_mem = MySubscription->workmem;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 536722b..ebe0423 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -45,17 +45,45 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -64,11 +92,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -84,16 +118,26 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names, int *logical_decoding_work_mem)
+ List **publication_names, int *logical_decoding_work_mem,
+ bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
bool work_mem_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -162,6 +206,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
*logical_decoding_work_mem = (int)parsed;
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -174,6 +235,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -197,7 +259,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
&data->publication_names,
- &logical_decoding_work_mem);
+ &logical_decoding_work_mem,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -217,6 +280,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -284,9 +368,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -312,19 +435,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -333,6 +463,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -361,14 +495,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -378,7 +512,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -387,7 +521,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -413,6 +547,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -433,13 +571,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -513,6 +652,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -549,6 +773,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*/
static RelationSyncEntry *
@@ -623,6 +875,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -657,7 +939,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index bb69683..3085c0f 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 9c06374..63fc2c7 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -969,6 +969,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 3394379..18f416f 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -50,6 +50,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
int32 subworkmem; /* Memory to use to decode changes. */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -76,6 +78,7 @@ typedef struct Subscription
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
int workmem; /* Memory to decode changes. */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e5a5d02..bbc9112 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2cc2dc4..ade4188 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out, TransactionId xid);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 66e89f0..1e4269c 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -163,6 +163,7 @@ typedef struct
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
int work_mem; /* Memory limit to use for decoding */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v7-0009-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From b1a405a2f2408015cb402fd6604579662ea58a4e Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Jan 2020 09:45:27 +0530
Subject: [PATCH v7 09/12] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dcb5811..180ea88 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1996,6 +1996,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 773edf8..cb9e6ee 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -785,7 +785,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 8e4744f..16515d0 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3264,6 +3268,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 63fc2c7..d7f22ae 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1293,7 +1293,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1314,7 +1314,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2357,6 +2358,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3196,7 +3200,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3253,6 +3257,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3276,6 +3283,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3362,6 +3372,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3610,11 +3625,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 427faa3..9ef4fbf 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5173,9 +5173,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 0510d38..7259c66 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -524,15 +524,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f..3888b0c 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 62eaf90..2dcb063 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1960,9 +1960,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v7-0007-Support-logical_decoding_work_mem-set-from-create.patch (application/octet-stream, 13.1K) download
From 307c8bf57e04bf93faadc1c7f2bef6368e5dbd0b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 11:51:04 +0530
Subject: [PATCH v7 07/12] Support logical_decoding_work_mem set from create
subscription command
---
doc/src/sgml/config.sgml | 21 +++++++++++
doc/src/sgml/ref/create_subscription.sgml | 12 ++++++
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 44 +++++++++++++++++++---
.../libpqwalreceiver/libpqwalreceiver.c | 3 ++
src/backend/replication/logical/worker.c | 1 +
src/backend/replication/pgoutput/pgoutput.c | 30 ++++++++++++++-
src/include/catalog/pg_subscription.h | 3 ++
src/include/replication/walreceiver.h | 1 +
9 files changed, 108 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5d1c902..8b1923c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1751,6 +1751,27 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-logical-decoding-work-mem" xreflabel="logical_decoding_work_mem">
+ <term><varname>logical_decoding_work_mem</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>logical_decoding_work_mem</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to be used by logical decoding,
+ before some of the decoded changes are either written to local disk.
+ This limits the amount of memory used by logical streaming replication
+ connections. It defaults to 64 megabytes (<literal>64MB</literal>).
+ Since each replication connection only uses a single buffer of this size,
+ and an installation normally doesn't have many such connections
+ concurrently (as limited by <varname>max_wal_senders</varname>), it's
+ safe to set this value significantly higher than <varname>work_mem</varname>,
+ reducing the amount of decoded changes written to disk.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..91790b0 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>work_mem</literal> (<type>integer</type>)</term>
+ <listitem>
+ <para>
+ Limits the amount of memory used to decode changes on the
+ publisher. If not specified, the publisher will use the default
+ specified by <varname>logical_decoding_work_mem</varname>. When
+ needed, additional data are spilled to disk.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index f77a83b..5cd1daa 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->workmem = subform->subworkmem;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 9bfe142..c50e854 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, int *logical_wm,
+ bool *logical_wm_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (logical_wm)
+ *logical_wm_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "work_mem") == 0 && logical_wm)
+ {
+ if (*logical_wm_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *logical_wm_given = true;
+ *logical_wm = defGetInt32(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ int logical_wm;
+ bool logical_wm_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &logical_wm, &logical_wm_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,12 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (logical_wm_given)
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ else
+ nulls[Anum_pg_subscription_subworkmem - 1] = true;
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -668,10 +689,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ int logical_wm;
+ bool logical_wm_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &logical_wm, &logical_wm_given);
if (slotname_given)
{
@@ -696,6 +720,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (logical_wm_given)
+ {
+ values[Anum_pg_subscription_subworkmem - 1] =
+ Int32GetDatum(logical_wm);
+ replaces[Anum_pg_subscription_subworkmem - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -707,7 +738,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -745,7 +777,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -782,7 +814,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 658af71..099d21b 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -406,6 +406,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ appendStringInfo(&cmd, ", work_mem '%d'",
+ options->proto.logical.work_mem);
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7a5471f..48b960c 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -1745,6 +1745,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.work_mem = MySubscription->workmem;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 7525082..536722b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -18,6 +18,7 @@
#include "replication/logicalproto.h"
#include "replication/origin.h"
#include "replication/pgoutput.h"
+#include "utils/guc.h"
#include "utils/int8.h"
#include "utils/inval.h"
#include "utils/memutils.h"
@@ -87,11 +88,12 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, int *logical_decoding_work_mem)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool work_mem_given = false;
foreach(lc, options)
{
@@ -137,6 +139,29 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "work_mem") == 0)
+ {
+ int64 parsed;
+
+ if (work_mem_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ work_mem_given = true;
+
+ if (!scanint8(strVal(defel->arg), true, &parsed))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid work_mem")));
+
+ if (parsed > PG_INT32_MAX || parsed < 64)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("work_mem \"%s\" out of range",
+ strVal(defel->arg))));
+
+ *logical_decoding_work_mem = (int)parsed;
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -171,7 +196,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &logical_decoding_work_mem);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..3394379 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ int32 subworkmem; /* Memory to use to decode changes. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ int workmem; /* Memory to decode changes. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index a276237..66e89f0 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -162,6 +162,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ int work_mem; /* Memory limit to use for decoding */
} logical;
} proto;
} WalRcvStreamOptions;
--
1.8.3.1
v7-0010-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From dfc942975061012a931debb4fccb466b38d7c029 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v7 10/12] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 6da7f71..086d0c7 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -70,7 +70,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v7-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup.patch (application/octet-stream, 1013B) download
From 4bd9c9773941c5b645e793f55d3063d06c9ce7ac Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:14:45 +0200
Subject: [PATCH v7 11/12] BUGFIX: set final_lsn for subxacts before cleanup
---
src/backend/replication/logical/reorderbuffer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 16515d0..c0c6a7f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -1327,6 +1327,10 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+ /* make sure subtxn has final_lsn */
+ if (subtxn->final_lsn == InvalidXLogRecPtr)
+ subtxn->final_lsn = txn->final_lsn;
+
/*
* Subtransactions are always associated to the toplevel TXN, even if
* they originally were happening inside another subtxn, so we won't
--
1.8.3.1
v7-0012-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From ed4c3d83ea72c05f949772d6b379838536bb14df Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v7 12/12] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 21:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-10 21:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-14 05:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-21 16:36 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2020-01-21 16:36 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Jan 14, 2020 at 10:56:37AM +0530, Dilip Kumar wrote:
>On Sat, Jan 11, 2020 at 3:07 AM Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>>
>> On 2020-Jan-10, Alvaro Herrera wrote:
>>
>> > Here's a rebase of this patch series. I didn't change anything except
>>
>> ... this time with attachments ...
>The patch set fails to apply on the head so rebased. (Rebased on
>commit cebf9d6e6ee13cbf9f1a91ec633cf96780ffc985)
>
I've noticed the patch was in WoA state since 2019/12/01, but there's
been quite a lot of traffic on this thread and a bunch of new patch
versions. So I've switched it to "needs review" - if that's not the
right status, let me know.
Also, the patch was moved forward mostly by Amit and Dilip, so I've
added them as authors in the CF app (well, what matters is the commit
message, of course, but let's keep this up to date too).
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-30 10:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 04:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2020-01-30 10:41 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Jan 10, 2020 at 10:14 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
>
> >
> > Few more comments:
> > --------------------------------
> > v4-0007-Implement-streaming-mode-in-ReorderBuffer
> > 1.
> > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > {
> > ..
> > + /*
> > + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> > + * information about
> > subtransactions, which could arrive after streaming start.
> > + */
> > + if (!txn->is_schema_sent)
> > + snapshot_now
> > = ReorderBufferCopySnap(rb, txn->base_snapshot,
> > + txn,
> > command_id);
> > ..
> > }
> >
> > Why are we using base snapshot here instead of the snapshot we saved
> > the first time streaming has happened? And as mentioned in comments,
> > won't we need to consider the snapshots for subtransactions that
> > arrived after the last time we have streamed the changes?
> Fixed
>
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
..
+ /*
+ * We can not use txn->snapshot_now directly because after we there
+ * might be some new sub-transaction which after the last streaming run
+ * so we need to add those sub-xip in the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
"because after we there", you seem to forget a word between 'we' and
'there'. So as we are copying it now, does this mean it will consider
the snapshots for subtransactions that arrived after the last time we
have streamed the changes? If so, have you tested it and can we add
the same in comments.
Also, if we need to copy the snapshot here, then do we need to again
copy it in ReorderBufferProcessTXN(in below code and in catch block in
the same function).
{
..
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
..
}
> >
> > 4. In ReorderBufferStreamTXN(), don't we need to set some of the txn
> > fields like origin_id, origin_lsn as we do in ReorderBufferCommit()
> > especially to cover the case when it gets called due to memory
> > overflow (aka via ReorderBufferCheckMemoryLimit).
> We get origin_lsn during commit time so I am not sure how can we do
> that. I have also noticed that currently, we are not using origin_lsn
> on the subscriber side. I think need more investigation that if we
> want this then do we need to log it early.
>
Have you done any investigation of this point? You might want to look
at pg_replication_origin* APIs. Today, again looking at this code, I
think with current coding, it won't be used even when we encounter
commit record. Because ReorderBufferCommit calls
ReorderBufferStreamCommit which will make sure that origin_id and
origin_lsn is never sent. I think at least that should be fixed, if
not, probably, we need a comment with reasoning why we think it is
okay not to do in this case.
+ /*
+ * If we are streaming the in-progress transaction then Discard the
/Discard/discard
> >
> > v4-0006-Gracefully-handle-concurrent-aborts-of-uncommitte
> > 1.
> > + /*
> > + * If CheckXidAlive is valid, then we check if it aborted. If it did, we
> > + * error out
> > + */
> > + if (TransactionIdIsValid(CheckXidAlive) &&
> > + !TransactionIdIsInProgress(CheckXidAlive) &&
> > + !TransactionIdDidCommit(CheckXidAlive))
> > + ereport(ERROR,
> > + (errcode(ERRCODE_TRANSACTION_ROLLBACK),
> > + errmsg("transaction aborted during system catalog scan")));
> >
> > Why here we can't use TransactionIdDidAbort? If we can't use it, then
> > can you add comments stating the reason of the same.
> Done
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. Instead of directly checking the abort status we do check
+ * if it is not in progress transaction and no committed. Because if there
+ * were a system crash then status of the the transaction which were running
+ * at that time might not have marked. So we need to consider them as
+ * aborted. Refer detailed comments at snapmgr.c where the variable is
+ * declared.
How about replacing the above comment with below one:
If CheckXidAlive is valid, then we check if it aborted. If it did, we
error out. We can't directly use TransactionIdDidAbort as after crash
such transaction might not have been marked as aborted. See detailed
comments at snapmgr.c where the variable is declared.
I am not able to understand the change in
v8-0011-BUGFIX-set-final_lsn-for-subxacts-before-cleanup. Do you have
any explanation for the same?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-01-30 12:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-01 05:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-01-30 12:39 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 30, 2020 at 4:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jan 10, 2020 at 10:14 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Mon, Jan 6, 2020 at 2:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> >
> > >
> > > Few more comments:
> > > --------------------------------
> > > v4-0007-Implement-streaming-mode-in-ReorderBuffer
> > > 1.
> > > +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> > > {
> > > ..
> > > + /*
> > > + * TOCHECK: We have to rebuild historic snapshot to be sure it includes all
> > > + * information about
> > > subtransactions, which could arrive after streaming start.
> > > + */
> > > + if (!txn->is_schema_sent)
> > > + snapshot_now
> > > = ReorderBufferCopySnap(rb, txn->base_snapshot,
> > > + txn,
> > > command_id);
> > > ..
> > > }
> > >
> > > Why are we using base snapshot here instead of the snapshot we saved
> > > the first time streaming has happened? And as mentioned in comments,
> > > won't we need to consider the snapshots for subtransactions that
> > > arrived after the last time we have streamed the changes?
> > Fixed
> >
>
> +static void
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> {
> ..
> + /*
> + * We can not use txn->snapshot_now directly because after we there
> + * might be some new sub-transaction which after the last streaming run
> + * so we need to add those sub-xip in the snapshot.
> + */
> + snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
> + txn, command_id);
>
> "because after we there", you seem to forget a word between 'we' and
> 'there'. So as we are copying it now, does this mean it will consider
> the snapshots for subtransactions that arrived after the last time we
> have streamed the changes? If so, have you tested it and can we add
> the same in comments.
Ok
> Also, if we need to copy the snapshot here, then do we need to again
> copy it in ReorderBufferProcessTXN(in below code and in catch block in
> the same function).
I think so because as part of the
"REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT" change, we might directly
point to the snapshot and that will get truncated when we truncate all
the changes of the ReorderBufferTXN. So I think we can check if
snapshot_now->copied is true then we can avoid copying otherwise we
can copy?
Other comments look fine to me so I will reply to them along with the
next version of the patch.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-01-31 02:38 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-01-31 02:38 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Jan 30, 2020 at 6:10 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Jan 30, 2020 at 4:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > Also, if we need to copy the snapshot here, then do we need to again
> > copy it in ReorderBufferProcessTXN(in below code and in catch block in
> > the same function).
> I think so because as part of the
> "REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT" change, we might directly
> point to the snapshot and that will get truncated when we truncate all
> the changes of the ReorderBufferTXN. So I think we can check if
> snapshot_now->copied is true then we can avoid copying otherwise we
> can copy?
>
Yeah, that makes sense, but I think then we also need to ensure that
ReorderBufferStreamTXN frees the snapshot only when it is copied. It
seems to me it should be always copied in the place where we are
trying to free it, so probably we should have an Assert there.
One more thing:
ReorderBufferProcessTXN()
{
..
+ if (streaming)
+ {
+ /*
+ * While streaming an in-progress transaction there is a
+ * possibility that the (sub)transaction might get aborted
+ * concurrently. In such case if the (sub)transaction has
+ * catalog update then we might decode the tuple using wrong
+ * catalog version. So for detecting the concurrent abort we
+ * set CheckXidAlive to the current (sub)transaction's xid for
+ * which this change belongs to. And, during catalog scan we
+ * can check the status of the xid and if it is aborted we will
+ * report an specific error which we can ignore. We might have
+ * already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the
+ * abort we will stream abort message to truncate the changes in
+ * the subscriber.
+ */
+ CheckXidAlive = change->txn->xid;
+ }
..
}
I think it is better to move the above code into an inline function
(something like SetXidAlive). It will make the code in function
ReorderBufferProcessTXN look cleaner and easier to understand.
> Other comments look fine to me so I will reply to them along with the
> next version of the patch.
>
Okay, thanks.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-02-05 04:15 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-02-05 04:15 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Jan 31, 2020 at 8:08 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jan 30, 2020 at 6:10 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Jan 30, 2020 at 4:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > Also, if we need to copy the snapshot here, then do we need to again
> > > copy it in ReorderBufferProcessTXN(in below code and in catch block in
> > > the same function).
> > I think so because as part of the
> > "REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT" change, we might directly
> > point to the snapshot and that will get truncated when we truncate all
> > the changes of the ReorderBufferTXN. So I think we can check if
> > snapshot_now->copied is true then we can avoid copying otherwise we
> > can copy?
> >
>
> Yeah, that makes sense, but I think then we also need to ensure that
> ReorderBufferStreamTXN frees the snapshot only when it is copied. It
> seems to me it should be always copied in the place where we are
> trying to free it, so probably we should have an Assert there.
>
> One more thing:
> ReorderBufferProcessTXN()
> {
> ..
> + if (streaming)
> + {
> + /*
> + * While streaming an in-progress transaction there is a
> + * possibility that the (sub)transaction might get aborted
> + * concurrently. In such case if the (sub)transaction has
> + * catalog update then we might decode the tuple using wrong
> + * catalog version. So for detecting the concurrent abort we
> + * set CheckXidAlive to the current (sub)transaction's xid for
> + * which this change belongs to. And, during catalog scan we
> + * can check the status of the xid and if it is aborted we will
> + * report an specific error which we can ignore. We might have
> + * already streamed some of the changes for the aborted
> + * (sub)transaction, but that is fine because when we decode the
> + * abort we will stream abort message to truncate the changes in
> + * the subscriber.
> + */
> + CheckXidAlive = change->txn->xid;
> + }
> ..
> }
>
> I think it is better to move the above code into an inline function
> (something like SetXidAlive). It will make the code in function
> ReorderBufferProcessTXN look cleaner and easier to understand.
>
Fixed in the latest version sent upthread.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-02-05 10:35 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-07 10:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Amit Kapila @ 2020-02-05 10:35 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Feb 5, 2020 at 9:46 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> Fixed in the latest version sent upthread.
>
Okay, thanks. I haven't looked at the latest version of patch series
as I was reviewing the previous version and I think all of these
comments are in the patch which is not modified. Here are my
comments:
I think we don't need to maintain
v8-0007-Support-logical_decoding_work_mem-set-from-create as per
discussion in one of the above emails [1] as its usage is not clear.
v8-0008-Add-support-for-streaming-to-built-in-replication
1.
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal>, <literal>work_mem</literal>
+ and <literal>streaming</literal>.
As per the discussion above [1], I don't think we need work_mem here.
You might want to remove the other usage from the patch as well.
2.
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool
*connect, bool *enabled_given,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
bool *refresh, int *logical_wm,
- bool *logical_wm_given)
+ bool *logical_wm_given, bool *streaming,
+ bool *streaming_given)
It is not clear to me why we need two parameters 'streaming' and
'streaming_given' in this API. Can't we handle similar to parameter
'refresh'?
3.
diff --git a/src/backend/replication/logical/launcher.c
b/src/backend/replication/logical/launcher.c
index aec885e..e80d00c 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,6 +14,8 @@
*
*-------------------------------------------------------------------------
*/
+#include <sys/types.h>
+#include <unistd.h>
#include "postgres.h"
I see only the above change in launcher.c. Why we need to include
these if there is no other change (at least not in this patch).
4.
stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache,
ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
Don't we want to set txn->final_lsn in report location as we do at few
other places?
5.
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
+ pq_sendbyte(out, 'D'); /* action DELETE */
+
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
- pq_sendbyte(out, 'D'); /* action DELETE */
Why this patch need to change the above code?
6.
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendint32(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgint(in, 4) == 1);
+
+ return xid;
+}
In these functions for sending bool, pq_sendint32 is used. Can't we
use pq_sendbyte similar to what we do in boolsend?
7.
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
In comments, 'starting to stream' is mentioned whereas this function
is to stop it.
8.
+void
+logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+}
+
+TransactionId
+logicalrep_read_stream_stop(StringInfo in)
+{
+ TransactionId xid;
+
+ xid = pq_getmsgint(in, 4);
+
+ return xid;
+}
Is there a reason to send xid on stopping stream? I don't see any use
of function logicalrep_read_stream_stop.
9.
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
{
..
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
..
+ pgstat_report_wait_end();
..
}
I see the calls to pgstat_report_wait_start/pgstat_report_wait_end in
this function, so not sure if the above comment makes sense.
10.
+ * The files are placed in /tmp by default, and the filenames include both
+ * the XID of the toplevel transaction and OID of the subscription.
Are we keeping files in /tmp or pg's temp tablespace dir. Seeing
below code, it doesn't seem that we place them in /tmp. If I am
correct, then can you update the comment.
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
11.
+ * The change is serialied in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
..
+ */
+static void
+stream_write_change(char action, StringInfo s)
The part of the comment which says "with length (not including the
length) .." is not clear to me. What does "not including the length"
mean?
12.
+ * TODO: Add missing_ok flag to specify in which cases it's OK not to
+ * find the files, and when it's an error.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid)
I think we can implement this TODO. It is clear when this function is
called from apply_handle_stream_commit, the file must exist. We can
similarly analyze other callers of this API.
13.
+apply_handle_stream_abort(StringInfo s)
{
..
+ /* FIXME optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
..
I am not sure how important this optimization is, so instead of FIXME,
it is better to keep it as a XXX comment. In the future, if we hit
any performance issue due to this, we can revisit our decision.
[1] - https://www.postgresql.org/message-id/CAA4eK1LH7xzF%2B-qHRv9EDXQTFYjPUYZw5B7FSK9QLEg7F603OQ%40mail.g...
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-02-07 10:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-10 08:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-02-07 10:48 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Feb 5, 2020 at 4:05 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Feb 5, 2020 at 9:46 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > Fixed in the latest version sent upthread.
> >
>
> Okay, thanks. I haven't looked at the latest version of patch series
> as I was reviewing the previous version and I think all of these
> comments are in the patch which is not modified. Here are my
> comments:
>
> I think we don't need to maintain
> v8-0007-Support-logical_decoding_work_mem-set-from-create as per
> discussion in one of the above emails [1] as its usage is not clear.
>
> v8-0008-Add-support-for-streaming-to-built-in-replication
> 1.
> - information. The allowed options are <literal>slot_name</literal> and
> - <literal>synchronous_commit</literal>
> + information. The allowed options are <literal>slot_name</literal>,
> + <literal>synchronous_commit</literal>, <literal>work_mem</literal>
> + and <literal>streaming</literal>.
>
> As per the discussion above [1], I don't think we need work_mem here.
> You might want to remove the other usage from the patch as well.
After putting more thought on this it appears that there could be some
use cases for setting the work_mem from the subscription, Assume a
case where data are coming from two different origins and based on the
origin ids different slots might collect different type of changes,
So isn't it good to have different work_mem for different slots? I am
not saying that the current way of implementing is the best one but
that we can improve. First, we need to decide whether we have a use
case for this or not. Please let me know your thought on the same.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-07 10:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-02-10 08:22 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-10 09:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-02-10 08:22 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Feb 7, 2020 at 4:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Feb 5, 2020 at 4:05 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Wed, Feb 5, 2020 at 9:46 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > Fixed in the latest version sent upthread.
> > >
> >
> > Okay, thanks. I haven't looked at the latest version of patch series
> > as I was reviewing the previous version and I think all of these
> > comments are in the patch which is not modified. Here are my
> > comments:
> >
> > I think we don't need to maintain
> > v8-0007-Support-logical_decoding_work_mem-set-from-create as per
> > discussion in one of the above emails [1] as its usage is not clear.
> >
> > v8-0008-Add-support-for-streaming-to-built-in-replication
> > 1.
> > - information. The allowed options are <literal>slot_name</literal> and
> > - <literal>synchronous_commit</literal>
> > + information. The allowed options are <literal>slot_name</literal>,
> > + <literal>synchronous_commit</literal>, <literal>work_mem</literal>
> > + and <literal>streaming</literal>.
> >
> > As per the discussion above [1], I don't think we need work_mem here.
> > You might want to remove the other usage from the patch as well.
>
> After putting more thought on this it appears that there could be some
> use cases for setting the work_mem from the subscription, Assume a
> case where data are coming from two different origins and based on the
> origin ids different slots might collect different type of changes,
> So isn't it good to have different work_mem for different slots? I am
> not saying that the current way of implementing is the best one but
> that we can improve. First, we need to decide whether we have a use
> case for this or not.
>
That is the whole point. I don't see a very clear usage of this and
neither did anybody explained clearly how it will be useful. I am not
denying that what you are describing has no use, but as you said we
might need to invent an entirely new way even if we have such a use.
I think it is better to avoid the requirements which are not essential
for this patch.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-07 10:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-10 08:22 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-02-10 09:39 ` Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-02-10 09:39 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Feb 10, 2020 at 1:52 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Feb 7, 2020 at 4:18 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Wed, Feb 5, 2020 at 4:05 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Wed, Feb 5, 2020 at 9:46 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > > Fixed in the latest version sent upthread.
> > > >
> > >
> > > Okay, thanks. I haven't looked at the latest version of patch series
> > > as I was reviewing the previous version and I think all of these
> > > comments are in the patch which is not modified. Here are my
> > > comments:
> > >
> > > I think we don't need to maintain
> > > v8-0007-Support-logical_decoding_work_mem-set-from-create as per
> > > discussion in one of the above emails [1] as its usage is not clear.
> > >
> > > v8-0008-Add-support-for-streaming-to-built-in-replication
> > > 1.
> > > - information. The allowed options are <literal>slot_name</literal> and
> > > - <literal>synchronous_commit</literal>
> > > + information. The allowed options are <literal>slot_name</literal>,
> > > + <literal>synchronous_commit</literal>, <literal>work_mem</literal>
> > > + and <literal>streaming</literal>.
> > >
> > > As per the discussion above [1], I don't think we need work_mem here.
> > > You might want to remove the other usage from the patch as well.
> >
> > After putting more thought on this it appears that there could be some
> > use cases for setting the work_mem from the subscription, Assume a
> > case where data are coming from two different origins and based on the
> > origin ids different slots might collect different type of changes,
> > So isn't it good to have different work_mem for different slots? I am
> > not saying that the current way of implementing is the best one but
> > that we can improve. First, we need to decide whether we have a use
> > case for this or not.
> >
>
> That is the whole point. I don't see a very clear usage of this and
> neither did anybody explained clearly how it will be useful. I am not
> denying that what you are describing has no use, but as you said we
> might need to invent an entirely new way even if we have such a use.
> I think it is better to avoid the requirements which are not essential
> for this patch.
Ok, I will include this change in the next patch set.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-02-11 03:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-02-11 03:12 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Feb 5, 2020 at 4:05 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Feb 5, 2020 at 9:46 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> I think we don't need to maintain
> v8-0007-Support-logical_decoding_work_mem-set-from-create as per
> discussion in one of the above emails [1] as its usage is not clear.
Done
> v8-0008-Add-support-for-streaming-to-built-in-replication
> 1.
> - information. The allowed options are <literal>slot_name</literal> and
> - <literal>synchronous_commit</literal>
> + information. The allowed options are <literal>slot_name</literal>,
> + <literal>synchronous_commit</literal>, <literal>work_mem</literal>
> + and <literal>streaming</literal>.
>
> As per the discussion above [1], I don't think we need work_mem here.
> You might want to remove the other usage from the patch as well.
Done
> 2.
> @@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool
> *connect, bool *enabled_given,
> bool *slot_name_given, char **slot_name,
> bool *copy_data, char **synchronous_commit,
> bool *refresh, int *logical_wm,
> - bool *logical_wm_given)
> + bool *logical_wm_given, bool *streaming,
> + bool *streaming_given)
>
> It is not clear to me why we need two parameters 'streaming' and
> 'streaming_given' in this API. Can't we handle similar to parameter
> 'refresh'?
The streaming option we need to update in the system table, so if we
don't remember whether the user has given its value or not then how we
will know whether to update this column or not? Or you are suggesting
that we should always mark this as updated but IMHO that is not a good
idea.
> 3.
> diff --git a/src/backend/replication/logical/launcher.c
> b/src/backend/replication/logical/launcher.c
> index aec885e..e80d00c 100644
> --- a/src/backend/replication/logical/launcher.c
> +++ b/src/backend/replication/logical/launcher.c
> @@ -14,6 +14,8 @@
> *
> *-------------------------------------------------------------------------
> */
> +#include <sys/types.h>
> +#include <unistd.h>
>
> #include "postgres.h"
>
> I see only the above change in launcher.c. Why we need to include
> these if there is no other change (at least not in this patch).
Removed
> 4.
> stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
> /* Push callback + info on the error context stack */
> state.ctx = ctx;
> state.callback_name = "stream_start";
> - /* state.report_location = apply_lsn; */
> + state.report_location = InvalidXLogRecPtr;
> errcallback.callback = output_plugin_error_callback;
> errcallback.arg = (void *) &state;
> errcallback.previous = error_context_stack;
> @@ -1194,7 +1194,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache,
> ReorderBufferTXN *txn)
> /* Push callback + info on the error context stack */
> state.ctx = ctx;
> state.callback_name = "stream_stop";
> - /* state.report_location = apply_lsn; */
> + state.report_location = InvalidXLogRecPtr;
> errcallback.callback = output_plugin_error_callback;
> errcallback.arg = (void *) &state;
> errcallback.previous = error_context_stack;
>
> Don't we want to set txn->final_lsn in report location as we do at few
> other places?
Fixed
> 5.
> -logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
> +logicalrep_write_delete(StringInfo out, TransactionId xid,
> + Relation rel, HeapTuple oldtuple)
> {
> + pq_sendbyte(out, 'D'); /* action DELETE */
> +
> Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
> rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
> rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
>
> - pq_sendbyte(out, 'D'); /* action DELETE */
>
> Why this patch need to change the above code?
Fixed
> 6.
> +void
> +logicalrep_write_stream_start(StringInfo out,
> + TransactionId xid, bool first_segment)
> +{
> + pq_sendbyte(out, 'S'); /* action STREAM START */
> +
> + Assert(TransactionIdIsValid(xid));
> +
> + /* transaction ID (we're starting to stream, so must be valid) */
> + pq_sendint32(out, xid);
> +
> + /* 1 if this is the first streaming segment for this xid */
> + pq_sendint32(out, first_segment ? 1 : 0);
> +}
> +
> +TransactionId
> +logicalrep_read_stream_start(StringInfo in, bool *first_segment)
> +{
> + TransactionId xid;
> +
> + Assert(first_segment);
> +
> + xid = pq_getmsgint(in, 4);
> + *first_segment = (pq_getmsgint(in, 4) == 1);
> +
> + return xid;
> +}
>
> In these functions for sending bool, pq_sendint32 is used. Can't we
> use pq_sendbyte similar to what we do in boolsend?
Done
> 7.
> +void
> +logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
> +{
> + pq_sendbyte(out, 'E'); /* action STREAM END */
> +
> + Assert(TransactionIdIsValid(xid));
> +
> + /* transaction ID (we're starting to stream, so must be valid) */
> + pq_sendint32(out, xid);
> +}
>
> In comments, 'starting to stream' is mentioned whereas this function
> is to stop it.
Fixed
> 8.
> +void
> +logicalrep_write_stream_stop(StringInfo out, TransactionId xid)
> +{
> + pq_sendbyte(out, 'E'); /* action STREAM END */
> +
> + Assert(TransactionIdIsValid(xid));
> +
> + /* transaction ID (we're starting to stream, so must be valid) */
> + pq_sendint32(out, xid);
> +}
> +
> +TransactionId
> +logicalrep_read_stream_stop(StringInfo in)
> +{
> + TransactionId xid;
> +
> + xid = pq_getmsgint(in, 4);
> +
> + return xid;
> +}
>
> Is there a reason to send xid on stopping stream? I don't see any use
> of function logicalrep_read_stream_stop.
Removed
> 9.
> + * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
> + */
> +static void
> +subxact_info_write(Oid subid, TransactionId xid)
> {
> ..
> + pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
> ..
> + pgstat_report_wait_end();
> ..
> }
>
> I see the calls to pgstat_report_wait_start/pgstat_report_wait_end in
> this function, so not sure if the above comment makes sense.
Fixed
> 10.
> + * The files are placed in /tmp by default, and the filenames include both
> + * the XID of the toplevel transaction and OID of the subscription.
>
> Are we keeping files in /tmp or pg's temp tablespace dir. Seeing
> below code, it doesn't seem that we place them in /tmp. If I am
> correct, then can you update the comment.
> +static void
> +subxact_filename(char *path, Oid subid, TransactionId xid)
> +{
> + char tempdirpath[MAXPGPATH];
> +
> + TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
Done
> 11.
> + * The change is serialied in a simple format, with length (not including
> + * the length), action code (identifying the message type) and message
> + * contents (without the subxact TransactionId value).
> + *
> ..
> + */
> +static void
> +stream_write_change(char action, StringInfo s)
>
> The part of the comment which says "with length (not including the
> length) .." is not clear to me. What does "not including the length"
> mean?
Basically, it says that the 4 bytes which are used for storing then
the length of total data doesn't include the 4 bytes.
> 12.
> + * TODO: Add missing_ok flag to specify in which cases it's OK not to
> + * find the files, and when it's an error.
> + */
> +static void
> +stream_cleanup_files(Oid subid, TransactionId xid)
>
> I think we can implement this TODO. It is clear when this function is
> called from apply_handle_stream_commit, the file must exist. We can
> similarly analyze other callers of this API.
Done
> 13.
> +apply_handle_stream_abort(StringInfo s)
> {
> ..
> + /* FIXME optimize the search by bsearch on sorted data */
> + for (i = nsubxacts; i > 0; i--)
> ..
>
> I am not sure how important this optimization is, so instead of FIXME,
> it is better to keep it as a XXX comment. In the future, if we hit
> any performance issue due to this, we can revisit our decision.
Done
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v10-0002-Issue-individual-invalidations-with-wal_level-lo.patch (application/octet-stream, 16.4K) download
From 4c2e923c86919b6e681bde0ec25a876402da2c19 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v10 02/10] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 50 +++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 ++++++++
src/backend/replication/logical/reorderbuffer.c | 55 +++++++++++++++---
src/backend/utils/cache/inval.c | 75 +++++++++++++++++++++++++
src/include/access/xact.h | 18 +++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 234 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index fbc5942..6191060 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -396,6 +401,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -423,7 +436,44 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index da32a4f..c9a64bb 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -5997,6 +5997,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a99fcaf..13a11ac 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index aeebbf2..3d6cbcf 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -473,6 +473,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1822,17 +1823,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2212,6 +2218,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2658,6 +2696,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2765,6 +2804,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3050,6 +3090,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..e0d04b9 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write individual invalidations into WAL to support
+ * the decoding of the in-progress transaction. As of now it was enough to
+ * log invalidation only at commit because we are only decoding the transaction
+ * at the commit time. We only need to log the catalog cache and relcache
+ * invalidation. There can not be any active MVCC scan in logical decoding so
+ * we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +499,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +523,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +545,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +565,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1552,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..6f2a583 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 626ecf4..9a3f045 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -459,6 +470,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v10-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From c73b48b0d2f023be736b53fdce13f73ceb10c4be Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v10 03/10] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 93c9488..64f651f 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -540,3 +569,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 8db9686..ace21ec 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index e3da7d3..ec40755 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -860,6 +908,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 9a3f045..15bb5ed 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -356,6 +356,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -395,6 +441,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v10-0005-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 37.8K) download
From 6ed52d0d4ab1c17c0f423c478f670c0efa3a98fe Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v10 05/10] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 710 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 36 ++
3 files changed, 693 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3ca960c..f68b2e4 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -769,6 +782,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -864,6 +909,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -987,7 +1035,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1023,6 +1071,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1037,6 +1088,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1320,6 +1374,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1345,8 +1408,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1354,9 +1502,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1495,63 +1640,75 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+/*
+ * Set xid for concurrent abort check
+ *
+ * While streaming an in-progress transaction there is a possibility that the
+ * (sub)transaction might get aborted concurrently. In such case if the
+ * (sub)transaction has catalog update then we might decode the tuple using
+ * wrong catalog version. So for detecting the concurrent abort we set
+ * CheckXidAlive to the current (sub)transaction's xid for which this change
+ * belongs to. And, during catalog scan we can check the status of the xid and
+ * if it is aborted we will report an specific error which we can ignore. We
+ * might have already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the abort we will
+ * stream abort message to truncate the changes in the subscriber.
+ */
+static inline void
+SetupCheckXidLive(TransactionId xid)
+{
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
+ if (!TransactionIdDidCommit(xid))
+ CheckXidAlive = xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
- snapshot_now = txn->base_snapshot;
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
- /* build data to be able to lookup the CommandIds of catalog tuples */
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1567,15 +1724,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1583,6 +1745,19 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* Set the xid for concurrent abort check. */
+ if (streaming)
+ SetupCheckXidLive(change->txn->xid);
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1592,8 +1767,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1659,7 +1832,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1680,8 +1861,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1699,7 +1878,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1757,7 +1936,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1766,10 +1953,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1800,9 +1993,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +2015,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1860,14 +2054,46 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ if (!XLogRecPtrIsInvalid(prev_lsn))
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1885,14 +2111,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,18 +2145,122 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1946,6 +2284,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2015,6 +2360,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2150,8 +2502,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2159,6 +2520,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2170,19 +2532,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2211,6 +2582,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2300,6 +2672,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2404,6 +2783,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2423,15 +2834,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2734,6 +3176,102 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after the last
+ * streaming run, we might have got some new sub-transactions. So we
+ * need to add them to the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ Assert(txn->snapshot_now->copied);
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 15bb5ed..adb8f9d 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -192,6 +193,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -227,6 +246,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -257,6 +286,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v10-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch (application/octet-stream, 12.6K) download
From 7ef2ab25181de245718bc12e3ae565aa8c62f3fa Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v10 04/10] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 40 ++++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++++--
src/include/utils/snapmgr.h | 4 ++-
6 files changed, 115 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index ace21ec..319349a 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index db6fad7..24d0d7a 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1304,6 +1304,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1423,6 +1432,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1537,6 +1554,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1686,6 +1711,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5483,6 +5516,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index c16eb05..5b0ef72 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,19 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +527,19 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +666,19 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3d6cbcf..3ca960c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -692,7 +692,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1551,7 +1551,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1802,7 +1802,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +1822,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v10-0008-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From 34a5848db37e831048142b0657ca7aea26af60fa Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v10 08/10] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 6da7f71..086d0c7 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -70,7 +70,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v10-0009-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 995f6e4f47b697a85290a1ef9dead428e67c2e46 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v10 09/10] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v10-0006-Add-support-for-streaming-to-built-in-replicatio.patch (application/octet-stream, 89.9K) download
From 941068d2af88d83f3bbb3ed9e0e81b32f9758e40 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 10 Feb 2020 11:00:48 +0530
Subject: [PATCH v10 06/10] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 45 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/launcher.c | 1 -
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 140 ++-
src/backend/replication/logical/worker.c | 1028 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 309 +++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2034 insertions(+), 39 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 6dfb2e4..47b414c 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -163,8 +163,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..3349cc4 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index f77a83b..7d7f721 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 119a9ce..54ca2d3 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,13 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -668,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -696,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -707,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -745,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -782,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 7169509..eb00242 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9..5257ab0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -408,6 +408,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..8156a42 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,7 +14,6 @@
*
*-------------------------------------------------------------------------
*/
-
#include "postgres.h"
#include "access/heapam.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index ec40755..b5d854f 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1146,7 +1146,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1191,7 +1191,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = txn->final_lsn;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd..5242ac0 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,10 +139,15 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -178,8 +183,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -187,6 +192,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -248,7 +257,8 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -256,6 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -296,6 +310,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -305,6 +320,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -347,12 +366,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -397,7 +420,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -405,6 +428,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -685,3 +712,104 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendbyte(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgbyte(in) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7a5471f..61388bd 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in tmp file directory by default, and the filenames
+ * include both the XID of the toplevel transaction and OID of the subscription.
+ * This is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -529,6 +636,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found PG_USED_FOR_ASSERTS_ONLY = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* XXX optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid, false);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -695,6 +1123,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -830,6 +1261,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -929,6 +1363,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1020,6 +1457,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1117,6 +1570,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i], true);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1132,6 +1601,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1580,6 +2052,561 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * missing_ok - don't report error for missing file is the flag is passed true.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid, true);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialized in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1745,6 +2772,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 7525082..11e249e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -44,17 +44,45 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -63,11 +91,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -83,15 +117,24 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -137,6 +180,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -149,6 +209,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -171,7 +232,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -191,6 +253,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -258,9 +341,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -286,19 +408,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -307,6 +436,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -335,14 +468,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -352,7 +485,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -361,7 +494,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -387,6 +520,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -407,13 +544,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -487,6 +625,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -523,6 +746,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*/
static RelationSyncEntry *
@@ -597,6 +848,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -631,7 +912,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 2c9d5de..30db2c2 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index abb533b..6ee7fa2 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -968,6 +968,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..617b909 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index aecb601..146d7c4 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2cc2dc4..277e44c 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e08afc6..0ebd140 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -169,6 +169,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v10-0007-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From 444ef04d5d2c4620d098de0926da40405893c94d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Jan 2020 09:45:27 +0530
Subject: [PATCH v10 07/10] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 8839699..2c7089e 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2004,6 +2004,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index c9e6060..d6f07d6 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -786,7 +786,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f68b2e4..a3c4509 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3267,6 +3271,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 6ee7fa2..21c4da0 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1292,7 +1292,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1313,7 +1313,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2356,6 +2357,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3194,7 +3198,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3251,6 +3255,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3274,6 +3281,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3360,6 +3370,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3608,11 +3623,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 2228256..969fa9e 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5193,9 +5193,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index adb8f9d..5e1337e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -519,15 +519,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f..3888b0c 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2ab2115..eed0088 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1983,9 +1983,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v10-0010-Bugfix-handling-of-incomplete-toast-tuple.patch (application/octet-stream, 13.3K) download
From 80bdec86bb5c2b9183e082e51558a7a81430a75c Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 30 Jan 2020 14:21:04 +0530
Subject: [PATCH v10 10/10] Bugfix handling of incomplete toast tuple
---
src/backend/access/heap/heapam.c | 3 +
src/backend/replication/logical/decode.c | 17 ++-
src/backend/replication/logical/reorderbuffer.c | 145 +++++++++++++-----------
src/include/access/heapam_xlog.h | 1 +
src/include/replication/reorderbuffer.h | 17 ++-
5 files changed, 110 insertions(+), 73 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 24d0d7a..faaaf67 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2018,6 +2018,9 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
{
xlrec.flags |= XLH_INSERT_CONTAINS_NEW_TUPLE;
bufflags |= REGBUF_KEEP_DATA;
+
+ if (IsToastRelation(relation))
+ xlrec.flags |= XLH_INSERT_ON_TOAST_RELATION;
}
XLogBeginInsert();
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 13a11ac..4ee528f 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -734,7 +734,9 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change,
+ xlrec->flags & XLH_INSERT_ON_TOAST_RELATION);
}
/*
@@ -801,7 +803,8 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -858,7 +861,8 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -894,7 +898,7 @@ DecodeTruncate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
memcpy(change->data.truncate.relids, xlrec->relids,
xlrec->nrelids * sizeof(Oid));
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
}
/*
@@ -999,7 +1003,7 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = false;
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
/* move to the next xl_multi_insert_tuple entry */
data += datalen;
@@ -1037,7 +1041,8 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index a3c4509..0cadc28 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -650,7 +650,7 @@ ReorderBufferTXNByXid(ReorderBuffer *rb, TransactionId xid, bool create,
*/
void
ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
- ReorderBufferChange *change)
+ ReorderBufferChange *change, bool toast_insert)
{
ReorderBufferTXN *txn;
@@ -664,6 +664,28 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If this is a toast insert then set the corresponding bit. Otherwise, if
+ * we have toast insert bit set and this is insert/update then clear the
+ * bit.
+ */
+ if (toast_insert)
+ txn->txn_flags |= RBTXN_HAS_TOAST_INSERT;
+ else if (rbtxn_has_toast_insert(txn) &&
+ ((change->action == REORDER_BUFFER_CHANGE_INSERT) ||
+ (change->action == REORDER_BUFFER_CHANGE_UPDATE)))
+ txn->txn_flags &= ~RBTXN_HAS_TOAST_INSERT;
+
+ /*
+ * If this is a speculative insert then set the corresponding bit.
+ * Otherwise, if we have speculative insert bit set and this is spec confirm
+ * record then clear the bit.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT)
+ txn->txn_flags |= RBTXN_HAS_SPEC_INSERT;
+ else if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM)
+ txn->txn_flags &= ~RBTXN_HAS_SPEC_INSERT;
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, true);
@@ -696,7 +718,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
change->data.msg.message = palloc(message_size);
memcpy(change->data.msg.message, message, message_size);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -1866,8 +1888,8 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* disk.
*/
dlist_delete(&change->node);
- ReorderBufferToastAppendChunk(rb, txn, relation,
- change);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
}
change_done:
@@ -2453,7 +2475,7 @@ ReorderBufferAddSnapshot(ReorderBuffer *rb, TransactionId xid,
change->data.snapshot = snap;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2502,7 +2524,7 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
change->data.command_id = cid;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2525,6 +2547,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
Size sz;
ReorderBufferTXN *txn;
+ ReorderBufferTXN *toptxn = NULL;
Assert(change->txn);
@@ -2540,7 +2563,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
/* if subxact, and streaming supported, use the toplevel instead */
if (txn->toptxn && ReorderBufferCanStream(rb))
- txn = txn->toptxn;
+ toptxn = txn->toptxn;
sz = ReorderBufferChangeSize(change);
@@ -2548,12 +2571,16 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
txn->size += sz;
rb->size += sz;
+ if (toptxn)
+ toptxn->size += sz;
}
else
{
Assert((rb->size >= sz) && (txn->size >= sz));
txn->size -= sz;
rb->size -= sz;
+ if (toptxn)
+ toptxn->size -= sz;
}
Assert(txn->size <= rb->size);
@@ -2619,7 +2646,7 @@ ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
change->data.inval.msg = msg;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -2806,15 +2833,16 @@ ReorderBufferLargestTopTXN(ReorderBuffer *rb)
txn = dlist_container(ReorderBufferTXN, node, iter.cur);
- /* if the current transaction is larger, remember it */
- if ((!largest) || (txn->size > largest->size))
- largest = txn;
+ /*
+ * if the current transaction is larger and doesn't have incomplete data
+ * remember it.
+ */
+ if (((!largest) || (txn->size > largest->size)) &&
+ ((txn->size > 0) && !rbtxn_has_toast_insert(txn) &&
+ !rbtxn_has_spec_insert(txn)))
+ largest = txn;
}
- Assert(largest);
- Assert(largest->size > 0);
- Assert(largest->size <= rb->size);
-
return largest;
}
@@ -2832,66 +2860,51 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
ReorderBufferTXN *txn;
- /* bail out if we haven't exceeded the memory limit */
- if (rb->size < logical_decoding_work_mem * 1024L)
- return;
-
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by streaming, if supported. Otherwise spill to disk.
- */
- if (ReorderBufferCanStream(rb))
+ /* Loop until we reach under the memory limit. */
+ while (rb->size >= logical_decoding_work_mem * 1024L)
{
/*
- * Pick the largest toplevel transaction and evict it from memory by
- * streaming the already decoded part.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTopTXN(rb);
+ if (ReorderBufferCanStream(rb) &&
+ (txn = ReorderBufferLargestTopTXN(rb)) != NULL)
+ {
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- /* we know there has to be one, because the size is not zero */
- Assert(txn && !txn->toptxn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
- ReorderBufferStreamTXN(rb, txn);
- }
- else
- {
/*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
- txn = ReorderBufferLargestTXN(rb);
-
- /* we know there has to be one, because the size is not zero */
- Assert(txn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
-
- ReorderBufferSerializeTXN(rb, txn);
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
}
- /*
- * After eviction, the transaction should have no entries in memory, and
- * should use 0 bytes for changes.
- *
- * XXX Checking the size is fine for both cases - spill to disk and
- * streaming. But for streaming we should really check nentries_mem for
- * all subtransactions too.
- */
- Assert(txn->size == 0);
- Assert(txn->nentries_mem == 0);
-
- /*
- * And furthermore, evicting the transaction should get us below the
- * memory limit again - it is not possible that we're still exceeding the
- * memory limit after evicting the transaction.
- *
- * This follows from the simple fact that the selected transaction is at
- * least as large as the most recent change (which caused us to go over
- * the memory limit). So by evicting it we're definitely back below the
- * memory limit.
- */
Assert(rb->size < logical_decoding_work_mem * 1024L);
}
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 95d18cd..aa17f7d 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -67,6 +67,7 @@
#define XLH_INSERT_LAST_IN_MULTI (1<<1)
#define XLH_INSERT_IS_SPECULATIVE (1<<2)
#define XLH_INSERT_CONTAINS_NEW_TUPLE (1<<3)
+#define XLH_INSERT_ON_TOAST_RELATION (1<<4)
/*
* xl_heap_update flag values, 8 bits are available.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5e1337e..a2646c5 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -174,6 +174,8 @@ typedef struct ReorderBufferChange
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
#define RBTXN_IS_STREAMED 0x0008
+#define RBTXN_HAS_TOAST_INSERT 0x0010
+#define RBTXN_HAS_SPEC_INSERT 0x0020
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -193,6 +195,17 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/* This transaction's changes has toast insert, without main table insert. */
+#define rbtxn_has_toast_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_TOAST_INSERT) != 0 \
+
+/*
+ * This transaction's changes has speculative insert, without speculative
+ * confirm.
+ */
+#define rbtxn_has_spec_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_SPEC_INSERT) != 0 \
+
/*
* Has this transaction been streamed to downstream?
*
@@ -547,7 +560,9 @@ void ReorderBufferReturnChange(ReorderBuffer *, ReorderBufferChange *);
Oid *ReorderBufferGetRelids(ReorderBuffer *, int nrelids);
void ReorderBufferReturnRelids(ReorderBuffer *, Oid *relids);
-void ReorderBufferQueueChange(ReorderBuffer *, TransactionId, XLogRecPtr lsn, ReorderBufferChange *);
+void ReorderBufferQueueChange(ReorderBuffer *, TransactionId,
+ XLogRecPtr lsn, ReorderBufferChange *,
+ bool incomplte_data);
void ReorderBufferQueueMessage(ReorderBuffer *, TransactionId, Snapshot snapshot, XLogRecPtr lsn,
bool transactional, const char *prefix,
Size message_size, const char *message);
--
1.8.3.1
v10-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.3K) download
From 9e9840335d75f6fae6236dda243b5a3ba925463b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v10 01/10] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index e3c60f2..da32a4f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -5998,3 +6000,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 2fa0a7f..b11b0c2 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 32f0225..51b6485 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1186,6 +1186,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1224,6 +1225,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 5e1dc8a..a99fcaf 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 98b033f..e23892a 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 4582196..756f6df 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -150,6 +150,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -285,6 +287,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-02-13 03:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-02-13 03:12 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Feb 11, 2020 at 8:42 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
The patch set was not applying on the head so I have rebased it.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v11-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.3K) download
From 50809a61c8ae64fa844954da4d5f0dca6a7a4f85 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v11 01/10] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index e3c60f2..da32a4f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -5998,3 +6000,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 2fa0a7f..b11b0c2 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 32f0225..51b6485 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1186,6 +1186,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1224,6 +1225,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 5e1dc8a..a99fcaf 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 98b033f..e23892a 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 4582196..756f6df 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -150,6 +150,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -285,6 +287,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v11-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch (application/octet-stream, 12.6K) download
From bf22b8b25df23514a1f33144ac1fc0d4cc2880d4 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v11 04/10] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 40 ++++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++++--
src/include/utils/snapmgr.h | 4 ++-
6 files changed, 115 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 3a95fb2..3a54b35 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index db6fad7..24d0d7a 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1304,6 +1304,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1423,6 +1432,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1537,6 +1554,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1686,6 +1711,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5483,6 +5516,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index c16eb05..5b0ef72 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -477,6 +478,19 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -513,6 +527,19 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -639,6 +666,19 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3d6cbcf..3ca960c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -692,7 +692,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1551,7 +1551,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1802,7 +1802,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +1822,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v11-0002-Issue-individual-invalidations-with-wal_level-lo.patch (application/octet-stream, 16.4K) download
From 47db44bc1ee6aac744eeef66537ca2023ad74c87 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v11 02/10] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 50 +++++++++++++++++
src/backend/access/transam/xact.c | 7 +++
src/backend/replication/logical/decode.c | 23 ++++++++
src/backend/replication/logical/reorderbuffer.c | 55 +++++++++++++++---
src/backend/utils/cache/inval.c | 75 +++++++++++++++++++++++++
src/include/access/xact.h | 18 +++++-
src/include/replication/reorderbuffer.h | 14 +++++
7 files changed, 234 insertions(+), 8 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index fbc5942..6191060 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,11 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -396,6 +401,14 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs,
+ xlrec->dbId, xlrec->tsId,
+ xlrec->relcacheInitFileInval);
+ }
}
const char *
@@ -423,7 +436,44 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs,
+ Oid dbId, Oid tsId,
+ bool relcacheInitFileInval)
+{
+ int i;
+
+ if (relcacheInitFileInval)
+ appendStringInfo(buf, "; relcache init file inval dbid %u tsid %u",
+ dbId, tsId);
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index da32a4f..c9a64bb 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -5997,6 +5997,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a99fcaf..13a11ac 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,29 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ /* XXX for now we're issuing invalidations one by one */
+ Assert(invals->nmsgs == 1);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->dbId, invals->tsId,
+ invals->relcacheInitFileInval,
+ invals->msgs[0]);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index aeebbf2..3d6cbcf 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -473,6 +473,7 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
@@ -1822,17 +1823,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation message locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ LocalExecuteInvalidationMessage(&change->data.inval.msg);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -2212,6 +2218,38 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.dbId = dbId;
+ change->data.inval.tsId = tsId;
+ change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
+ change->data.inval.msg = msg;
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2658,6 +2696,7 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -2765,6 +2804,7 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
/* ReorderBufferChange contains everything important */
break;
}
@@ -3050,6 +3090,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
break;
}
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..e0d04b9 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write individual invalidations into WAL to support
+ * the decoding of the in-progress transaction. As of now it was enough to
+ * log invalidation only at commit because we are only decoding the transaction
+ * at the commit time. We only need to log the catalog cache and relcache
+ * invalidation. There can not be any active MVCC scan in logical decoding so
+ * we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,9 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -489,6 +499,18 @@ RegisterCatcacheInvalidation(int cacheId,
{
AddCatcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
cacheId, hashValue, dbId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cc.id = (int8) cacheId;
+ msg.cc.dbId = dbId;
+ msg.cc.hashValue = hashValue;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -501,6 +523,18 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
{
AddCatalogInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, catId);
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.cat.id = SHAREDINVALCATALOG_ID;
+ msg.cat.dbId = dbId;
+ msg.cat.catId = catId;
+
+ LogLogicalInvalidations(1, &msg, false);
+ }
}
/*
@@ -511,6 +545,8 @@ RegisterCatalogInvalidation(Oid dbId, Oid catId)
static void
RegisterRelcacheInvalidation(Oid dbId, Oid relId)
{
+ bool RelcacheInitFileInval = false;
+
AddRelcacheInvalidationMessage(&transInvalInfo->CurrentCmdInvalidMsgs,
dbId, relId);
@@ -529,7 +565,22 @@ RegisterRelcacheInvalidation(Oid dbId, Oid relId)
* as well. Also zap when we are invalidating whole relcache.
*/
if (relId == InvalidOid || RelationIdIsInInitFile(relId))
+ {
transInvalInfo->RelcacheInitFileInval = true;
+ RelcacheInitFileInval = true;
+ }
+
+ /* Issue an invalidation WAL record (when wal_level=logical) */
+ if (XLogLogicalInfoActive())
+ {
+ SharedInvalidationMessage msg;
+
+ msg.rc.id = SHAREDINVALRELCACHE_ID;
+ msg.rc.dbId = dbId;
+ msg.rc.relId = relId;
+
+ LogLogicalInvalidations(1, &msg, RelcacheInitFileInval);
+ }
}
/*
@@ -1501,3 +1552,27 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations(int nmsgs, SharedInvalidationMessage *msgs,
+ bool relcacheInitFileInval)
+{
+ xl_xact_invalidations xlrec;
+
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.dbId = MyDatabaseId;
+ xlrec.tsId = MyDatabaseTableSpace;
+ xlrec.relcacheInitFileInval = relcacheInitFileInval;
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..6f2a583 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,22 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ *
+ * XXX Currently nmsgs=1 but that might change in the future.
+ */
+typedef struct xl_xact_invalidations
+{
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init file */
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 626ecf4..9a3f045 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,16 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ Oid dbId; /* MyDatabaseId */
+ Oid tsId; /* MyDatabaseTableSpace */
+ bool relcacheInitFileInval; /* invalidate relcache init
+ * file */
+ SharedInvalidationMessage msg; /* invalidation message */
+ } inval;
} data;
/*
@@ -459,6 +470,9 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ Oid dbId, Oid tsId, bool relcacheInitFileInval,
+ SharedInvalidationMessage msg);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v11-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From 1680c9678f93474063293a7051d8400afba97662 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v11 03/10] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 93c9488..64f651f 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -540,3 +569,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index bce6d37..3a95fb2 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index e3da7d3..ec40755 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -860,6 +908,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 9a3f045..15bb5ed 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -356,6 +356,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -395,6 +441,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v11-0005-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 37.8K) download
From 74072fe1077efa3bb7e689cb23bba285b6a254d3 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v11 05/10] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 710 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 36 ++
3 files changed, 693 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 3ca960c..f68b2e4 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -769,6 +782,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -864,6 +909,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -987,7 +1035,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1023,6 +1071,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1037,6 +1088,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1320,6 +1374,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1345,8 +1408,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1354,9 +1502,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1495,63 +1640,75 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+/*
+ * Set xid for concurrent abort check
+ *
+ * While streaming an in-progress transaction there is a possibility that the
+ * (sub)transaction might get aborted concurrently. In such case if the
+ * (sub)transaction has catalog update then we might decode the tuple using
+ * wrong catalog version. So for detecting the concurrent abort we set
+ * CheckXidAlive to the current (sub)transaction's xid for which this change
+ * belongs to. And, during catalog scan we can check the status of the xid and
+ * if it is aborted we will report an specific error which we can ignore. We
+ * might have already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the abort we will
+ * stream abort message to truncate the changes in the subscriber.
+ */
+static inline void
+SetupCheckXidLive(TransactionId xid)
+{
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
+ if (!TransactionIdDidCommit(xid))
+ CheckXidAlive = xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
- snapshot_now = txn->base_snapshot;
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
- /* build data to be able to lookup the CommandIds of catalog tuples */
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1567,15 +1724,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1583,6 +1745,19 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* Set the xid for concurrent abort check. */
+ if (streaming)
+ SetupCheckXidLive(change->txn->xid);
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1592,8 +1767,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1659,7 +1832,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1680,8 +1861,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1699,7 +1878,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1757,7 +1936,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1766,10 +1953,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1800,9 +1993,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1822,7 +2015,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1860,14 +2054,46 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ if (!XLogRecPtrIsInvalid(prev_lsn))
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1885,14 +2111,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,18 +2145,122 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1946,6 +2284,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2015,6 +2360,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2150,8 +2502,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2159,6 +2520,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2170,19 +2532,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2211,6 +2582,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2300,6 +2672,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2404,6 +2783,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2423,15 +2834,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2734,6 +3176,102 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after the last
+ * streaming run, we might have got some new sub-transactions. So we
+ * need to add them to the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ Assert(txn->snapshot_now->copied);
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 15bb5ed..adb8f9d 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -173,6 +173,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -192,6 +193,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -227,6 +246,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -257,6 +286,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v11-0007-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From 044d924ca93ca8adf88ccc388eb5b27299f10d60 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Tue, 11 Feb 2020 12:10:43 +0530
Subject: [PATCH v11 07/10] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 9129f79..ecf1c57 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2016,6 +2016,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f681aaf..2ede8f3 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -787,7 +787,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f68b2e4..a3c4509 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3267,6 +3271,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 6ee7fa2..21c4da0 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1292,7 +1292,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1313,7 +1313,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2356,6 +2357,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3194,7 +3198,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3251,6 +3255,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3274,6 +3281,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3360,6 +3370,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3608,11 +3623,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 226c904..7f95079 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5193,9 +5193,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index adb8f9d..5e1337e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -519,15 +519,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f..3888b0c 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 634f825..f9c30e8 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1984,9 +1984,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v11-0006-Add-support-for-streaming-to-built-in-replicatio.patch (application/octet-stream, 89.9K) download
From 1a15a960aa9eeb9f47173e1897866864394045bc Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 10 Feb 2020 11:00:48 +0530
Subject: [PATCH v11 06/10] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 45 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/launcher.c | 1 -
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 140 ++-
src/backend/replication/logical/worker.c | 1028 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 309 +++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2034 insertions(+), 39 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 1b8bead..95b7c24 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -164,8 +164,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..3349cc4 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index f77a83b..7d7f721 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 119a9ce..54ca2d3 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,13 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -668,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -696,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -707,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -745,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -782,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 7169509..eb00242 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4102,6 +4102,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9..5257ab0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -408,6 +408,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..8156a42 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,7 +14,6 @@
*
*-------------------------------------------------------------------------
*/
-
#include "postgres.h"
#include "access/heapam.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index ec40755..b5d854f 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1146,7 +1146,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1191,7 +1191,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = txn->final_lsn;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd..5242ac0 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,10 +139,15 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -178,8 +183,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -187,6 +192,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -248,7 +257,8 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -256,6 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -296,6 +310,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -305,6 +320,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -347,12 +366,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -397,7 +420,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -405,6 +428,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -685,3 +712,104 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendbyte(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgbyte(in) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 7a5471f..61388bd 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in tmp file directory by default, and the filenames
+ * include both the XID of the toplevel transaction and OID of the subscription.
+ * This is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -60,6 +82,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -67,6 +90,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -106,12 +130,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -163,6 +234,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -529,6 +636,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found PG_USED_FOR_ASSERTS_ONLY = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* XXX optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ /* We should not receive aborts for unknown subtransactions. */
+ Assert(found);
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid, false);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -541,6 +960,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -556,6 +978,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -591,6 +1016,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -695,6 +1123,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -830,6 +1261,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -929,6 +1363,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1020,6 +1457,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1117,6 +1570,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i], true);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1132,6 +1601,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1580,6 +2052,561 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * missing_ok - don't report error for missing file is the flag is passed true.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid, true);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialized in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1745,6 +2772,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 7525082..11e249e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -44,17 +44,45 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -63,11 +91,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -83,15 +117,24 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -137,6 +180,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -149,6 +209,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -171,7 +232,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -191,6 +253,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -258,9 +341,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -286,19 +408,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -307,6 +436,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -335,14 +468,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -352,7 +485,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -361,7 +494,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -387,6 +520,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -407,13 +544,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -487,6 +625,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -523,6 +746,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*/
static RelationSyncEntry *
@@ -597,6 +848,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -631,7 +912,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 2c9d5de..30db2c2 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index abb533b..6ee7fa2 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -968,6 +968,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..617b909 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index aecb601..146d7c4 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2cc2dc4..277e44c 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e08afc6..0ebd140 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -169,6 +169,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v11-0008-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From 2b5c0bcf4d6e9bb4896b5530595047de26167793 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v11 08/10] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 6da7f71..086d0c7 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -70,7 +70,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v11-0009-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 7ce58a6151eb1a3afe1a1510f3547d8c1cb8a27f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v11 09/10] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v11-0010-Bugfix-handling-of-incomplete-toast-tuple.patch (application/octet-stream, 13.3K) download
From 170dfd601dcb64d9969436cd6eba35d518280716 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 30 Jan 2020 14:21:04 +0530
Subject: [PATCH v11 10/10] Bugfix handling of incomplete toast tuple
---
src/backend/access/heap/heapam.c | 3 +
src/backend/replication/logical/decode.c | 17 ++-
src/backend/replication/logical/reorderbuffer.c | 145 +++++++++++++-----------
src/include/access/heapam_xlog.h | 1 +
src/include/replication/reorderbuffer.h | 17 ++-
5 files changed, 110 insertions(+), 73 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 24d0d7a..faaaf67 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2018,6 +2018,9 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
{
xlrec.flags |= XLH_INSERT_CONTAINS_NEW_TUPLE;
bufflags |= REGBUF_KEEP_DATA;
+
+ if (IsToastRelation(relation))
+ xlrec.flags |= XLH_INSERT_ON_TOAST_RELATION;
}
XLogBeginInsert();
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 13a11ac..4ee528f 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -734,7 +734,9 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change,
+ xlrec->flags & XLH_INSERT_ON_TOAST_RELATION);
}
/*
@@ -801,7 +803,8 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -858,7 +861,8 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -894,7 +898,7 @@ DecodeTruncate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
memcpy(change->data.truncate.relids, xlrec->relids,
xlrec->nrelids * sizeof(Oid));
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
}
/*
@@ -999,7 +1003,7 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = false;
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
/* move to the next xl_multi_insert_tuple entry */
data += datalen;
@@ -1037,7 +1041,8 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index a3c4509..0cadc28 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -650,7 +650,7 @@ ReorderBufferTXNByXid(ReorderBuffer *rb, TransactionId xid, bool create,
*/
void
ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
- ReorderBufferChange *change)
+ ReorderBufferChange *change, bool toast_insert)
{
ReorderBufferTXN *txn;
@@ -664,6 +664,28 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If this is a toast insert then set the corresponding bit. Otherwise, if
+ * we have toast insert bit set and this is insert/update then clear the
+ * bit.
+ */
+ if (toast_insert)
+ txn->txn_flags |= RBTXN_HAS_TOAST_INSERT;
+ else if (rbtxn_has_toast_insert(txn) &&
+ ((change->action == REORDER_BUFFER_CHANGE_INSERT) ||
+ (change->action == REORDER_BUFFER_CHANGE_UPDATE)))
+ txn->txn_flags &= ~RBTXN_HAS_TOAST_INSERT;
+
+ /*
+ * If this is a speculative insert then set the corresponding bit.
+ * Otherwise, if we have speculative insert bit set and this is spec confirm
+ * record then clear the bit.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT)
+ txn->txn_flags |= RBTXN_HAS_SPEC_INSERT;
+ else if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM)
+ txn->txn_flags &= ~RBTXN_HAS_SPEC_INSERT;
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, true);
@@ -696,7 +718,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
change->data.msg.message = palloc(message_size);
memcpy(change->data.msg.message, message, message_size);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -1866,8 +1888,8 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* disk.
*/
dlist_delete(&change->node);
- ReorderBufferToastAppendChunk(rb, txn, relation,
- change);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
}
change_done:
@@ -2453,7 +2475,7 @@ ReorderBufferAddSnapshot(ReorderBuffer *rb, TransactionId xid,
change->data.snapshot = snap;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2502,7 +2524,7 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
change->data.command_id = cid;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2525,6 +2547,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
Size sz;
ReorderBufferTXN *txn;
+ ReorderBufferTXN *toptxn = NULL;
Assert(change->txn);
@@ -2540,7 +2563,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
/* if subxact, and streaming supported, use the toplevel instead */
if (txn->toptxn && ReorderBufferCanStream(rb))
- txn = txn->toptxn;
+ toptxn = txn->toptxn;
sz = ReorderBufferChangeSize(change);
@@ -2548,12 +2571,16 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
txn->size += sz;
rb->size += sz;
+ if (toptxn)
+ toptxn->size += sz;
}
else
{
Assert((rb->size >= sz) && (txn->size >= sz));
txn->size -= sz;
rb->size -= sz;
+ if (toptxn)
+ toptxn->size -= sz;
}
Assert(txn->size <= rb->size);
@@ -2619,7 +2646,7 @@ ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
change->data.inval.relcacheInitFileInval = relcacheInitFileInval;
change->data.inval.msg = msg;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -2806,15 +2833,16 @@ ReorderBufferLargestTopTXN(ReorderBuffer *rb)
txn = dlist_container(ReorderBufferTXN, node, iter.cur);
- /* if the current transaction is larger, remember it */
- if ((!largest) || (txn->size > largest->size))
- largest = txn;
+ /*
+ * if the current transaction is larger and doesn't have incomplete data
+ * remember it.
+ */
+ if (((!largest) || (txn->size > largest->size)) &&
+ ((txn->size > 0) && !rbtxn_has_toast_insert(txn) &&
+ !rbtxn_has_spec_insert(txn)))
+ largest = txn;
}
- Assert(largest);
- Assert(largest->size > 0);
- Assert(largest->size <= rb->size);
-
return largest;
}
@@ -2832,66 +2860,51 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
ReorderBufferTXN *txn;
- /* bail out if we haven't exceeded the memory limit */
- if (rb->size < logical_decoding_work_mem * 1024L)
- return;
-
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by streaming, if supported. Otherwise spill to disk.
- */
- if (ReorderBufferCanStream(rb))
+ /* Loop until we reach under the memory limit. */
+ while (rb->size >= logical_decoding_work_mem * 1024L)
{
/*
- * Pick the largest toplevel transaction and evict it from memory by
- * streaming the already decoded part.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTopTXN(rb);
+ if (ReorderBufferCanStream(rb) &&
+ (txn = ReorderBufferLargestTopTXN(rb)) != NULL)
+ {
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- /* we know there has to be one, because the size is not zero */
- Assert(txn && !txn->toptxn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
- ReorderBufferStreamTXN(rb, txn);
- }
- else
- {
/*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
- txn = ReorderBufferLargestTXN(rb);
-
- /* we know there has to be one, because the size is not zero */
- Assert(txn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
-
- ReorderBufferSerializeTXN(rb, txn);
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
}
- /*
- * After eviction, the transaction should have no entries in memory, and
- * should use 0 bytes for changes.
- *
- * XXX Checking the size is fine for both cases - spill to disk and
- * streaming. But for streaming we should really check nentries_mem for
- * all subtransactions too.
- */
- Assert(txn->size == 0);
- Assert(txn->nentries_mem == 0);
-
- /*
- * And furthermore, evicting the transaction should get us below the
- * memory limit again - it is not possible that we're still exceeding the
- * memory limit after evicting the transaction.
- *
- * This follows from the simple fact that the selected transaction is at
- * least as large as the most recent change (which caused us to go over
- * the memory limit). So by evicting it we're definitely back below the
- * memory limit.
- */
Assert(rb->size < logical_decoding_work_mem * 1024L);
}
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 95d18cd..aa17f7d 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -67,6 +67,7 @@
#define XLH_INSERT_LAST_IN_MULTI (1<<1)
#define XLH_INSERT_IS_SPECULATIVE (1<<2)
#define XLH_INSERT_CONTAINS_NEW_TUPLE (1<<3)
+#define XLH_INSERT_ON_TOAST_RELATION (1<<4)
/*
* xl_heap_update flag values, 8 bits are available.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 5e1337e..a2646c5 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -174,6 +174,8 @@ typedef struct ReorderBufferChange
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
#define RBTXN_IS_STREAMED 0x0008
+#define RBTXN_HAS_TOAST_INSERT 0x0010
+#define RBTXN_HAS_SPEC_INSERT 0x0020
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -193,6 +195,17 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/* This transaction's changes has toast insert, without main table insert. */
+#define rbtxn_has_toast_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_TOAST_INSERT) != 0 \
+
+/*
+ * This transaction's changes has speculative insert, without speculative
+ * confirm.
+ */
+#define rbtxn_has_spec_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_SPEC_INSERT) != 0 \
+
/*
* Has this transaction been streamed to downstream?
*
@@ -547,7 +560,9 @@ void ReorderBufferReturnChange(ReorderBuffer *, ReorderBufferChange *);
Oid *ReorderBufferGetRelids(ReorderBuffer *, int nrelids);
void ReorderBufferReturnRelids(ReorderBuffer *, Oid *relids);
-void ReorderBufferQueueChange(ReorderBuffer *, TransactionId, XLogRecPtr lsn, ReorderBufferChange *);
+void ReorderBufferQueueChange(ReorderBuffer *, TransactionId,
+ XLogRecPtr lsn, ReorderBufferChange *,
+ bool incomplte_data);
void ReorderBufferQueueMessage(ReorderBuffer *, TransactionId, Snapshot snapshot, XLogRecPtr lsn,
bool transactional, const char *prefix,
Size message_size, const char *message);
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-02-29 05:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-02-29 05:07 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Feb 13, 2020 at 8:42 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Feb 11, 2020 at 8:42 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> The patch set was not applying on the head so I have rebased it.
I have changed the patch 0002 so that instead of logging the WAL for
each invalidation, now we log at each command end as discussed
upthread[1]
Soon we will evaluate the performance for the same and post the results.
[1] https://www.postgresql.org/message-id/CAA4eK1LOa%2B2KqNX%3Dm%3D1qMBDW%2Bo50AuwjAOX6ZqL-rWGiH1F9MQ%40...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v12-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.3K) download
From b092f8ab260a8616e093c5d075010a307eae41ee Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 09:53:08 +0530
Subject: [PATCH v12 01/10] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 45 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 ++++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 +++++++++++++--------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 98 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index e3c60f2..da32a4f 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -5096,6 +5097,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -5998,3 +6000,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 2fa0a7f..b11b0c2 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -88,11 +88,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -194,6 +196,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -397,7 +403,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -743,6 +749,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 32f0225..51b6485 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1186,6 +1186,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1224,6 +1225,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 5e1dc8a..a99fcaf 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ record->toplevel_xid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrIds) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 98b033f..e23892a 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -227,6 +227,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 4582196..756f6df 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -150,6 +150,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -285,6 +287,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v12-0002-Issue-individual-invalidations-with-wal_level-lo.patch (application/octet-stream, 17.9K) download
From 0e124734e61a36407a5907b9bd2d514d5b3764b3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v12 02/10] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 40 +++++++++
src/backend/access/transam/xact.c | 7 ++
src/backend/replication/logical/decode.c | 18 ++++
src/backend/replication/logical/reorderbuffer.c | 110 +++++++++++++++++++++---
src/backend/utils/cache/inval.c | 59 +++++++++++--
src/include/access/xact.h | 13 ++-
src/include/replication/reorderbuffer.h | 11 +++
7 files changed, 239 insertions(+), 19 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index fbc5942..17c06f7 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,9 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -396,6 +399,12 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs);
+ }
}
const char *
@@ -423,7 +432,38 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs)
+{
+ int i;
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index da32a4f..c9a64bb 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -5997,6 +5997,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index a99fcaf..693ac01 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,24 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->nmsgs, invals->msgs);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 481277a..03f6888 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -220,7 +220,7 @@ static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
-static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs);
/*
* ---------------------------------------
@@ -455,6 +455,11 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
pfree(change->data.msg.message);
change->data.msg.message = NULL;
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ if (change->data.inval.invalidations)
+ pfree(change->data.inval.invalidations);
+ change->data.inval.invalidations = NULL;
+ break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
if (change->data.snapshot)
{
@@ -1819,17 +1824,24 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation messages locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ ReorderBufferExecuteInvalidations(
+ change->data.inval.ninvalidations,
+ change->data.inval.invalidations);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -1871,7 +1883,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -1897,7 +1910,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2209,6 +2223,39 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, int nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ MemoryContextAlloc(rb->context,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2239,12 +2286,12 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
* in the changestream but we don't know which those are.
*/
static void
-ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs)
{
int i;
- for (i = 0; i < txn->ninvalidations; i++)
- LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+ for (i = 0; i < nmsgs; i++)
+ LocalExecuteInvalidationMessage(&msgs[i]);
}
/*
@@ -2596,6 +2643,24 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ {
+ char *data;
+ Size inval_size = sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+
+ sz += inval_size;
+
+ ReorderBufferSerializeReserve(rb, sz);
+ data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
+
+ /* might have been reallocated above */
+ ondisk = (ReorderBufferDiskChange *) rb->outbuf;
+ memcpy(data, change->data.inval.invalidations, inval_size);
+ data += inval_size;
+
+ break;
+ }
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot snap;
@@ -2741,6 +2806,11 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ sz += sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot snap;
@@ -3009,6 +3079,20 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ {
+ Size inval_size = sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+
+ change->data.inval.invalidations =
+ MemoryContextAlloc(rb->context,
+ change->data.msg.message_size);
+ /* read the message */
+ memcpy(change->data.inval.invalidations, data, inval_size);
+ data += inval_size;
+
+ break;
+ }
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot oldsnap;
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..47be680 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write invalidations into WAL at each command end to
+ * support the decoding of the in-progress transaction. As of now it was
+ * enough to log invalidation only at commit because we are only decoding the
+ * transaction at the commit time. We only need to log the catalog cache and
+ * relcache invalidation. There can not be any active MVCC scan in logical
+ * decoding so we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,8 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(void);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -1068,8 +1077,8 @@ AtEOSubXact_Inval(bool isCommit)
/*
* CommandEndInvalidationMessages
- * Process queued-up invalidation messages at end of one command
- * in a transaction.
+ * Process queued-up invalidation messages at end of one command
+ * in a transaction.
*
* Here, we send no messages to the shared queue, since we don't know yet if
* we will commit. We do need to locally process the CurrentCmdInvalidMsgs
@@ -1078,8 +1087,8 @@ AtEOSubXact_Inval(bool isCommit)
* of the prior-cmds list.
*
* Note:
- * This should be called during CommandCounterIncrement(),
- * after we have advanced the command ID.
+ * This should be called during CommandCounterIncrement(),
+ * after we have advanced the command ID.
*/
void
CommandEndInvalidationMessages(void)
@@ -1090,7 +1099,10 @@ CommandEndInvalidationMessages(void)
* just quietly return if no state to work on.
*/
if (transInvalInfo == NULL)
- return;
+ return;
+
+ if (XLogLogicalInfoActive())
+ LogLogicalInvalidations();
ProcessInvalidationMessages(&transInvalInfo->CurrentCmdInvalidMsgs,
LocalExecuteInvalidationMessage);
@@ -1501,3 +1513,40 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations()
+{
+ xl_xact_invalidations xlrec;
+ SharedInvalidationMessage *invalMessages;
+ int nmsgs = 0;
+
+ if (transInvalInfo->CurrentCmdInvalidMsgs.cclist)
+ {
+ ProcessInvalidationMessagesMulti(&transInvalInfo->CurrentCmdInvalidMsgs,
+ MakeSharedInvalidMessagesArray);
+ invalMessages = SharedInvalidMessagesArray;
+ nmsgs = numSharedInvalidMessagesArray;
+ SharedInvalidMessagesArray = NULL;
+ numSharedInvalidMessagesArray = 0;
+ }
+
+ if (nmsgs > 0)
+ {
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) invalMessages,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+
+ pfree(invalMessages);
+ }
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..b822c5e 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,17 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ */
+typedef struct xl_xact_invalidations
+{
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 626ecf4..af35287 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,14 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ uint32 ninvalidations; /* Number of messages */
+ SharedInvalidationMessage *invalidations; /* invalidation
+ * message */
+ } inval;
} data;
/*
@@ -459,6 +468,8 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ int nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v12-0005-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 37.8K) download
From ae386d1de96900d2b2277f2b13f45dcb937009ce Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v12 05/10] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 710 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 36 ++
3 files changed, 693 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index ddfba27..f18a31a 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -773,6 +786,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -865,6 +910,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -988,7 +1036,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1024,6 +1072,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1038,6 +1089,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1321,6 +1375,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1346,8 +1409,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1355,9 +1503,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1496,63 +1641,75 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+/*
+ * Set xid for concurrent abort check
+ *
+ * While streaming an in-progress transaction there is a possibility that the
+ * (sub)transaction might get aborted concurrently. In such case if the
+ * (sub)transaction has catalog update then we might decode the tuple using
+ * wrong catalog version. So for detecting the concurrent abort we set
+ * CheckXidAlive to the current (sub)transaction's xid for which this change
+ * belongs to. And, during catalog scan we can check the status of the xid and
+ * if it is aborted we will report an specific error which we can ignore. We
+ * might have already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the abort we will
+ * stream abort message to truncate the changes in the subscriber.
+ */
+static inline void
+SetupCheckXidLive(TransactionId xid)
+{
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
+ if (!TransactionIdDidCommit(xid))
+ CheckXidAlive = xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
- snapshot_now = txn->base_snapshot;
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
- /* build data to be able to lookup the CommandIds of catalog tuples */
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1568,15 +1725,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1584,6 +1746,19 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* Set the xid for concurrent abort check. */
+ if (streaming)
+ SetupCheckXidLive(change->txn->xid);
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1593,8 +1768,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1660,7 +1833,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1681,8 +1862,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1700,7 +1879,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1758,7 +1937,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1767,10 +1954,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1801,9 +1994,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1823,7 +2016,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1863,14 +2057,46 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ if (!XLogRecPtrIsInvalid(prev_lsn))
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1889,14 +2115,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1916,18 +2150,122 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1951,6 +2289,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2020,6 +2365,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2155,8 +2507,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2164,6 +2525,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2175,19 +2537,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2216,6 +2587,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2306,6 +2678,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2410,6 +2789,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2429,15 +2840,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2757,6 +3199,102 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after the last
+ * streaming run, we might have got some new sub-transactions. So we
+ * need to add them to the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ Assert(txn->snapshot_now->copied);
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e102840..6d65986 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -171,6 +171,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -190,6 +191,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -225,6 +244,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -255,6 +284,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v12-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From de2a3f48e527fd65c1ff7092a691070f0ff783f9 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v12 03/10] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 93c9488..64f651f 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -540,3 +569,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index bce6d37..3a95fb2 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index e3da7d3..ec40755 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -860,6 +908,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index af35287..e102840 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -354,6 +354,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -393,6 +439,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v12-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch (application/octet-stream, 12.6K) download
From fc8043cb461e77195a3d08279c7fd66b1309948a Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:32:34 +0530
Subject: [PATCH v12 04/10] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 40 ++++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++++--
src/include/utils/snapmgr.h | 4 ++-
6 files changed, 115 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 3a95fb2..3a54b35 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>txid_current()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 5a32e62..0a4d86d 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1304,6 +1304,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1423,6 +1432,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1537,6 +1554,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1686,6 +1711,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5491,6 +5524,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index dfba5ae..413a21f 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -481,6 +482,19 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -517,6 +531,19 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -643,6 +670,19 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 03f6888..ddfba27 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -696,7 +696,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1552,7 +1552,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1803,7 +1803,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1823,7 +1823,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v12-0007-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From 65365ef3ca81aa26cfd69da49224dfeb44bf605c Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Tue, 11 Feb 2020 12:10:43 +0530
Subject: [PATCH v12 07/10] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 87586a7..cb294b5 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2024,6 +2024,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f681aaf..2ede8f3 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -787,7 +787,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f18a31a..f1e7498 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3290,6 +3294,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 6ee7fa2..21c4da0 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1292,7 +1292,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1313,7 +1313,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2356,6 +2357,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3194,7 +3198,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3251,6 +3255,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3274,6 +3281,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3360,6 +3370,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3608,11 +3623,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 07a86c7..72ff008 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5196,9 +5196,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6d65986..603f325 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -517,15 +517,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f..3888b0c 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 634f825..f9c30e8 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1984,9 +1984,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_ssl| SELECT s.pid,
s.ssl,
--
1.8.3.1
v12-0006-Add-support-for-streaming-to-built-in-replicatio.patch (application/octet-stream, 89.8K) download
From 625e8fad4d4a1f9e7eb177d1f2edd7a3cb86cdf0 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 10 Feb 2020 11:00:48 +0530
Subject: [PATCH v12 06/10] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 45 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/launcher.c | 1 -
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 140 ++-
src/backend/replication/logical/worker.c | 1028 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 309 +++++-
src/backend/replication/slotfuncs.c | 7 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2034 insertions(+), 39 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 1b8bead..95b7c24 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -164,8 +164,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..3349cc4 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index f77a83b..7d7f721 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 119a9ce..54ca2d3 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -58,7 +58,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -89,6 +90,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -174,6 +177,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -317,6 +330,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -333,7 +348,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -411,6 +426,13 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -668,10 +690,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -696,6 +721,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -707,7 +739,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -745,7 +778,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -782,7 +815,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 462b4d7..32d85cb 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4105,6 +4105,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9..5257ab0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -408,6 +408,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..8156a42 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,7 +14,6 @@
*
*-------------------------------------------------------------------------
*/
-
#include "postgres.h"
#include "access/heapam.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index ec40755..b5d854f 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1146,7 +1146,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1191,7 +1191,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = txn->final_lsn;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd..5242ac0 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,10 +139,15 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -178,8 +183,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -187,6 +192,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -248,7 +257,8 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -256,6 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -296,6 +310,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -305,6 +320,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -347,12 +366,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -397,7 +420,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -405,6 +428,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -685,3 +712,104 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendbyte(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgbyte(in) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index ad4a732..cc89188 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in tmp file directory by default, and the filenames
+ * include both the XID of the toplevel transaction and OID of the subscription.
+ * This is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -31,6 +52,7 @@
#include "catalog/namespace.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -61,6 +83,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -68,6 +91,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -107,12 +131,59 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
+static bool handle_streamed_transaction(const char action, StringInfo s);
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
/*
* Should this worker apply changes for given relation.
*
@@ -164,6 +235,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -530,6 +637,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* XXX optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid, false);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -542,6 +961,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -557,6 +979,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -592,6 +1017,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -696,6 +1124,9 @@ apply_handle_update(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -833,6 +1264,9 @@ apply_handle_delete(StringInfo s)
bool found;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -932,6 +1366,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1023,6 +1460,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1120,6 +1573,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i], true);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1135,6 +1604,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1583,6 +2055,561 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * missing_ok - don't report error for missing file is the flag is passed true.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid, true);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialized in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -1748,6 +2775,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 7525082..11e249e 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -44,17 +44,45 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-/* Entry in the map used to remember which relation schemas we sent. */
+/*
+ * Entry in the map used to remember which relation schemas we sent.
+ *
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ */
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
+ TransactionId xid; /* transaction that created the record */
bool schema_sent; /* did we send the schema? */
+ List *streamed_txns; /* streamed toplevel transactions with
+ * this schema */
bool replicate_valid;
PublicationActions pubactions;
} RelationSyncEntry;
@@ -63,11 +91,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -83,15 +117,24 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -137,6 +180,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -149,6 +209,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -171,7 +232,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -191,6 +253,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -258,9 +341,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (!relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (!schema_sent)
{
TupleDesc desc;
int i;
@@ -286,19 +408,26 @@ maybe_send_schema(LogicalDecodingContext *ctx,
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
- relentry->schema_sent = true;
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -307,6 +436,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -335,14 +468,14 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INSERT:
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation,
+ logicalrep_write_insert(ctx->out, xid, relation,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -352,7 +485,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
&change->data.tp.oldtuple->tuple : NULL;
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple,
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple,
&change->data.tp.newtuple->tuple);
OutputPluginWrite(ctx, true);
break;
@@ -361,7 +494,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
if (change->data.tp.oldtuple)
{
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation,
+ logicalrep_write_delete(ctx->out, xid, relation,
&change->data.tp.oldtuple->tuple);
OutputPluginWrite(ctx, true);
}
@@ -387,6 +520,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -407,13 +544,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -487,6 +625,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -523,6 +746,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*/
static RelationSyncEntry *
@@ -597,6 +848,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -631,7 +912,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 2c9d5de..30db2c2 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -147,6 +147,13 @@ create_logical_replication_slot(char *name, char *plugin,
logical_read_local_xlog_page, NULL, NULL,
NULL);
+ /*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+
/* build initial snapshot, might take a while */
DecodingContextFindStartpoint(ctx);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index abb533b..6ee7fa2 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -968,6 +968,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..617b909 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 3a65a51..ed2e43a 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -945,7 +945,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 2cc2dc4..277e44c 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -85,25 +89,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e08afc6..0ebd140 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -169,6 +169,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v12-0008-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From 210dd338ba977043e2588c7ac9315d2ed1b40242 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v12 08/10] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 6da7f71..086d0c7 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -70,7 +70,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v12-0009-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 090eb648bbd5e5a0bbaa04dc61476009e8d3bec1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v12 09/10] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v12-0010-Bugfix-handling-of-incomplete-toast-tuple.patch (application/octet-stream, 13.3K) download
From 72bc3bb31c31ee94e6aca9577632155d6657e23f Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 28 Feb 2020 11:07:46 +0530
Subject: [PATCH v12 10/10] Bugfix handling of incomplete toast tuple
---
src/backend/access/heap/heapam.c | 3 +
src/backend/replication/logical/decode.c | 17 ++-
src/backend/replication/logical/reorderbuffer.c | 145 +++++++++++++-----------
src/include/access/heapam_xlog.h | 1 +
src/include/replication/reorderbuffer.h | 17 ++-
5 files changed, 110 insertions(+), 73 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0a4d86d..b3ab8c6 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2018,6 +2018,9 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
{
xlrec.flags |= XLH_INSERT_CONTAINS_NEW_TUPLE;
bufflags |= REGBUF_KEEP_DATA;
+
+ if (IsToastRelation(relation))
+ xlrec.flags |= XLH_INSERT_ON_TOAST_RELATION;
}
XLogBeginInsert();
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 693ac01..b7d507c 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -729,7 +729,9 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change,
+ xlrec->flags & XLH_INSERT_ON_TOAST_RELATION);
}
/*
@@ -796,7 +798,8 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -853,7 +856,8 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -889,7 +893,7 @@ DecodeTruncate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
memcpy(change->data.truncate.relids, xlrec->relids,
xlrec->nrelids * sizeof(Oid));
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
}
/*
@@ -994,7 +998,7 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = false;
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
/* move to the next xl_multi_insert_tuple entry */
data += datalen;
@@ -1032,7 +1036,8 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index f1e7498..b53417e 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -654,7 +654,7 @@ ReorderBufferTXNByXid(ReorderBuffer *rb, TransactionId xid, bool create,
*/
void
ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
- ReorderBufferChange *change)
+ ReorderBufferChange *change, bool toast_insert)
{
ReorderBufferTXN *txn;
@@ -668,6 +668,28 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If this is a toast insert then set the corresponding bit. Otherwise, if
+ * we have toast insert bit set and this is insert/update then clear the
+ * bit.
+ */
+ if (toast_insert)
+ txn->txn_flags |= RBTXN_HAS_TOAST_INSERT;
+ else if (rbtxn_has_toast_insert(txn) &&
+ ((change->action == REORDER_BUFFER_CHANGE_INSERT) ||
+ (change->action == REORDER_BUFFER_CHANGE_UPDATE)))
+ txn->txn_flags &= ~RBTXN_HAS_TOAST_INSERT;
+
+ /*
+ * If this is a speculative insert then set the corresponding bit.
+ * Otherwise, if we have speculative insert bit set and this is spec confirm
+ * record then clear the bit.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT)
+ txn->txn_flags |= RBTXN_HAS_SPEC_INSERT;
+ else if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM)
+ txn->txn_flags &= ~RBTXN_HAS_SPEC_INSERT;
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, true);
@@ -700,7 +722,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
change->data.msg.message = palloc(message_size);
memcpy(change->data.msg.message, message, message_size);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -1867,8 +1889,8 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* disk.
*/
dlist_delete(&change->node);
- ReorderBufferToastAppendChunk(rb, txn, relation,
- change);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
}
change_done:
@@ -2458,7 +2480,7 @@ ReorderBufferAddSnapshot(ReorderBuffer *rb, TransactionId xid,
change->data.snapshot = snap;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2507,7 +2529,7 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
change->data.command_id = cid;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2530,6 +2552,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
Size sz;
ReorderBufferTXN *txn;
+ ReorderBufferTXN *toptxn = NULL;
Assert(change->txn);
@@ -2545,7 +2568,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
/* if subxact, and streaming supported, use the toplevel instead */
if (txn->toptxn && ReorderBufferCanStream(rb))
- txn = txn->toptxn;
+ toptxn = txn->toptxn;
sz = ReorderBufferChangeSize(change);
@@ -2553,12 +2576,16 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
txn->size += sz;
rb->size += sz;
+ if (toptxn)
+ toptxn->size += sz;
}
else
{
Assert((rb->size >= sz) && (txn->size >= sz));
txn->size -= sz;
rb->size -= sz;
+ if (toptxn)
+ toptxn->size -= sz;
}
Assert(txn->size <= rb->size);
@@ -2625,7 +2652,7 @@ ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
memcpy(change->data.inval.invalidations, msgs,
sizeof(SharedInvalidationMessage) * nmsgs);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -2812,15 +2839,16 @@ ReorderBufferLargestTopTXN(ReorderBuffer *rb)
txn = dlist_container(ReorderBufferTXN, node, iter.cur);
- /* if the current transaction is larger, remember it */
- if ((!largest) || (txn->size > largest->size))
- largest = txn;
+ /*
+ * if the current transaction is larger and doesn't have incomplete data
+ * remember it.
+ */
+ if (((!largest) || (txn->size > largest->size)) &&
+ ((txn->size > 0) && !rbtxn_has_toast_insert(txn) &&
+ !rbtxn_has_spec_insert(txn)))
+ largest = txn;
}
- Assert(largest);
- Assert(largest->size > 0);
- Assert(largest->size <= rb->size);
-
return largest;
}
@@ -2838,66 +2866,51 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
ReorderBufferTXN *txn;
- /* bail out if we haven't exceeded the memory limit */
- if (rb->size < logical_decoding_work_mem * 1024L)
- return;
-
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by streaming, if supported. Otherwise spill to disk.
- */
- if (ReorderBufferCanStream(rb))
+ /* Loop until we reach under the memory limit. */
+ while (rb->size >= logical_decoding_work_mem * 1024L)
{
/*
- * Pick the largest toplevel transaction and evict it from memory by
- * streaming the already decoded part.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTopTXN(rb);
+ if (ReorderBufferCanStream(rb) &&
+ (txn = ReorderBufferLargestTopTXN(rb)) != NULL)
+ {
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- /* we know there has to be one, because the size is not zero */
- Assert(txn && !txn->toptxn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
- ReorderBufferStreamTXN(rb, txn);
- }
- else
- {
/*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
- txn = ReorderBufferLargestTXN(rb);
-
- /* we know there has to be one, because the size is not zero */
- Assert(txn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
-
- ReorderBufferSerializeTXN(rb, txn);
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
}
- /*
- * After eviction, the transaction should have no entries in memory, and
- * should use 0 bytes for changes.
- *
- * XXX Checking the size is fine for both cases - spill to disk and
- * streaming. But for streaming we should really check nentries_mem for
- * all subtransactions too.
- */
- Assert(txn->size == 0);
- Assert(txn->nentries_mem == 0);
-
- /*
- * And furthermore, evicting the transaction should get us below the
- * memory limit again - it is not possible that we're still exceeding the
- * memory limit after evicting the transaction.
- *
- * This follows from the simple fact that the selected transaction is at
- * least as large as the most recent change (which caused us to go over
- * the memory limit). So by evicting it we're definitely back below the
- * memory limit.
- */
Assert(rb->size < logical_decoding_work_mem * 1024L);
}
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 95d18cd..aa17f7d 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -67,6 +67,7 @@
#define XLH_INSERT_LAST_IN_MULTI (1<<1)
#define XLH_INSERT_IS_SPECULATIVE (1<<2)
#define XLH_INSERT_CONTAINS_NEW_TUPLE (1<<3)
+#define XLH_INSERT_ON_TOAST_RELATION (1<<4)
/*
* xl_heap_update flag values, 8 bits are available.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 603f325..9a4f886 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -172,6 +172,8 @@ typedef struct ReorderBufferChange
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
#define RBTXN_IS_STREAMED 0x0008
+#define RBTXN_HAS_TOAST_INSERT 0x0010
+#define RBTXN_HAS_SPEC_INSERT 0x0020
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -191,6 +193,17 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/* This transaction's changes has toast insert, without main table insert. */
+#define rbtxn_has_toast_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_TOAST_INSERT) != 0 \
+
+/*
+ * This transaction's changes has speculative insert, without speculative
+ * confirm.
+ */
+#define rbtxn_has_spec_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_SPEC_INSERT) != 0 \
+
/*
* Has this transaction been streamed to downstream?
*
@@ -545,7 +558,9 @@ void ReorderBufferReturnChange(ReorderBuffer *, ReorderBufferChange *);
Oid *ReorderBufferGetRelids(ReorderBuffer *, int nrelids);
void ReorderBufferReturnRelids(ReorderBuffer *, Oid *relids);
-void ReorderBufferQueueChange(ReorderBuffer *, TransactionId, XLogRecPtr lsn, ReorderBufferChange *);
+void ReorderBufferQueueChange(ReorderBuffer *, TransactionId,
+ XLogRecPtr lsn, ReorderBufferChange *,
+ bool incomplte_data);
void ReorderBufferQueueMessage(ReorderBuffer *, TransactionId, Snapshot snapshot, XLogRecPtr lsn,
bool transactional, const char *prefix,
Size message_size, const char *message);
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-03-03 21:46 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-03 21:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-04 04:58 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 3 replies; 536+ messages in thread
From: Tomas Vondra @ 2020-03-03 21:46 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
Hi,
I started looking at this patch series again, hoping to get it moving
for PG13. There's been a tremendous amount of work done since I last
worked on it, and a lot was discussed on this thread, so it'll take a
while to get familiar with the new code ...
The first thing I realized that WAL-logging of assignments in v12 does
both the "old" logging (using dedicated message) and "new" with
toplevel-XID embedded in the first message. Yes, the patch was wrong,
because it eliminated all calls to ProcArrayApplyXidAssignment() and so
it was trivial to crash the replica due to KnownAssignedXids overflow.
But I don't think re-introducing XLOG_XACT_ASSIGNMENT message is the
right fix.
I actually proposed doing this (having both ways to log assignments) so
that there's no regression risk with (wal_level < logical). But IIRC
Andres objected to it, argumenting that we should not log the same piece
of information in two very different ways at the same time (IIRC it was
discussed on the FOSDEM dev meeting, so I don't have a link to share).
And I do agree with him ...
The question is, why couldn't the replica use the same assignment info
we already write for logical decoding? The main challenge is that now
the assignment can be sent in many different xlog messages, from a bunch
of resource managers (essentially, any xlog message with a xid can have
embedded XID of the toplevel xact). So the handling would either need to
happen in every rmgr, or we need to move it before we call the rmgr.
For exampple, we might do this e.g. in StartupXLOG() I think, per the
attached patch (FWIW this particular fix was written by Masahiko Sawada,
not me). This does the trick for me - I'm no longer able to reproduce
the KnownAssignedXids overflow.
The one difference is that we used to call ProcArrayApplyXidAssignment
for larger groups of XIDs, as sent in the assignment message. Now we
call it for each individual assignment. I don't know if this is an
issue, but I suppose we might introduce some sort of local caching
(accumulate the assignments into a local array, call the function only
when we have enough of them).
Aside from that, I think there's a minor bug in xact.c - the patch adds
a "assigned" field to TransactionStateData, but then it fails to add a
default value into TopTransactionStateData. We probably interpret NULL
as false, but then there's nothing for the pointer. I suspect it might
leave some random garbage there, leading to strange things later.
Another thing I noticed is LogicalDecodingProcessRecord() extracts the
toplevel XID using a macro
txid = XLogRecGetTopXid(record);
but then it just starts accessing the fields directly again in the
ReorderBufferAssignChild call. I think we should do this instead:
ReorderBufferAssignChild(ctx->reorder,
txid,
XLogRecGetXid(record),
buf.origptr);
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2020-03-03 21:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2 siblings, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2020-03-03 21:59 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
D'oh! As usual I forgot to actually attach the patch I mentioned. So
here it is ...
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7574,6 +7574,19 @@ StartupXLOG(void)
LWLockRelease(XidGenLock);
}
+ /*
+ * Assign subtransaction xids to the top level xid if the
+ * record has that information. This is required at most
+ * once per subtransactions.
+ */
+ if (TransactionIdIsValid(xlogreader->toplevel_xid) &&
+ standbyState >= STANDBY_INITIALIZED)
+ {
+ Assert(XLogStandbyInfoActive());
+ ProcArrayApplyXidAssignment(xlogreader->toplevel_xid,
+ 1, &record->xl_xid);
+ }
+
/*
* Before replaying this record, check if this record causes
* the current timeline to change. The record is already
Attachments:
xid-assignment-v12-fix.patch (text/plain, 946B) download
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7574,6 +7574,19 @@ StartupXLOG(void)
LWLockRelease(XidGenLock);
}
+ /*
+ * Assign subtransaction xids to the top level xid if the
+ * record has that information. This is required at most
+ * once per subtransactions.
+ */
+ if (TransactionIdIsValid(xlogreader->toplevel_xid) &&
+ standbyState >= STANDBY_INITIALIZED)
+ {
+ Assert(XLogStandbyInfoActive());
+ ProcArrayApplyXidAssignment(xlogreader->toplevel_xid,
+ 1, &record->xl_xid);
+ }
+
/*
* Before replaying this record, check if this record causes
* the current timeline to change. The record is already
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2020-03-04 03:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-05 17:55 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2 siblings, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-03-04 03:43 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Mar 4, 2020 at 3:16 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> Hi,
>
> I started looking at this patch series again, hoping to get it moving
> for PG13.
Nice.
There's been a tremendous amount of work done since I last
> worked on it, and a lot was discussed on this thread, so it'll take a
> while to get familiar with the new code ...
>
> The first thing I realized that WAL-logging of assignments in v12 does
> both the "old" logging (using dedicated message) and "new" with
> toplevel-XID embedded in the first message. Yes, the patch was wrong,
> because it eliminated all calls to ProcArrayApplyXidAssignment() and so
> it was trivial to crash the replica due to KnownAssignedXids overflow.
> But I don't think re-introducing XLOG_XACT_ASSIGNMENT message is the
> right fix.
>
> I actually proposed doing this (having both ways to log assignments) so
> that there's no regression risk with (wal_level < logical). But IIRC
> Andres objected to it, argumenting that we should not log the same piece
> of information in two very different ways at the same time (IIRC it was
> discussed on the FOSDEM dev meeting, so I don't have a link to share).
> And I do agree with him ...
>
> The question is, why couldn't the replica use the same assignment info
> we already write for logical decoding? The main challenge is that now
> the assignment can be sent in many different xlog messages, from a bunch
> of resource managers (essentially, any xlog message with a xid can have
> embedded XID of the toplevel xact). So the handling would either need to
> happen in every rmgr, or we need to move it before we call the rmgr.
>
> For exampple, we might do this e.g. in StartupXLOG() I think, per the
> attached patch (FWIW this particular fix was written by Masahiko Sawada,
> not me). This does the trick for me - I'm no longer able to reproduce
> the KnownAssignedXids overflow.
>
> The one difference is that we used to call ProcArrayApplyXidAssignment
> for larger groups of XIDs, as sent in the assignment message. Now we
> call it for each individual assignment. I don't know if this is an
> issue, but I suppose we might introduce some sort of local caching
> (accumulate the assignments into a local array, call the function only
> when we have enough of them).
Thanks for the pointers, I will think over these points.
>
> Aside from that, I think there's a minor bug in xact.c - the patch adds
> a "assigned" field to TransactionStateData, but then it fails to add a
> default value into TopTransactionStateData. We probably interpret NULL
> as false, but then there's nothing for the pointer. I suspect it might
> leave some random garbage there, leading to strange things later.
Actually, we will never access that field for the
TopTransactionStateData, right?
See below code, we have a check that only if IsSubTransaction(), then
we access the "assigned" filed.
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
>
> Another thing I noticed is LogicalDecodingProcessRecord() extracts the
> toplevel XID using a macro
>
> txid = XLogRecGetTopXid(record);
>
> but then it just starts accessing the fields directly again in the
> ReorderBufferAssignChild call. I think we should do this instead:
>
> ReorderBufferAssignChild(ctx->reorder,
> txid,
> XLogRecGetXid(record),
> buf.origptr);
Make sense. I will change this in the patch.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-03-05 17:55 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 536+ messages in thread
From: Tomas Vondra @ 2020-03-05 17:55 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Mar 04, 2020 at 09:13:49AM +0530, Dilip Kumar wrote:
>On Wed, Mar 4, 2020 at 3:16 AM Tomas Vondra
><tomas.vondra@2ndquadrant.com> wrote:
>>
>> Hi,
>>
>> I started looking at this patch series again, hoping to get it moving
>> for PG13.
>
>Nice.
>
> There's been a tremendous amount of work done since I last
>> worked on it, and a lot was discussed on this thread, so it'll take a
>> while to get familiar with the new code ...
>>
>> The first thing I realized that WAL-logging of assignments in v12 does
>> both the "old" logging (using dedicated message) and "new" with
>> toplevel-XID embedded in the first message. Yes, the patch was wrong,
>> because it eliminated all calls to ProcArrayApplyXidAssignment() and so
>> it was trivial to crash the replica due to KnownAssignedXids overflow.
>> But I don't think re-introducing XLOG_XACT_ASSIGNMENT message is the
>> right fix.
>>
>> I actually proposed doing this (having both ways to log assignments) so
>> that there's no regression risk with (wal_level < logical). But IIRC
>> Andres objected to it, argumenting that we should not log the same piece
>> of information in two very different ways at the same time (IIRC it was
>> discussed on the FOSDEM dev meeting, so I don't have a link to share).
>> And I do agree with him ...
>>
>> The question is, why couldn't the replica use the same assignment info
>> we already write for logical decoding? The main challenge is that now
>> the assignment can be sent in many different xlog messages, from a bunch
>> of resource managers (essentially, any xlog message with a xid can have
>> embedded XID of the toplevel xact). So the handling would either need to
>> happen in every rmgr, or we need to move it before we call the rmgr.
>>
>> For exampple, we might do this e.g. in StartupXLOG() I think, per the
>> attached patch (FWIW this particular fix was written by Masahiko Sawada,
>> not me). This does the trick for me - I'm no longer able to reproduce
>> the KnownAssignedXids overflow.
>>
>> The one difference is that we used to call ProcArrayApplyXidAssignment
>> for larger groups of XIDs, as sent in the assignment message. Now we
>> call it for each individual assignment. I don't know if this is an
>> issue, but I suppose we might introduce some sort of local caching
>> (accumulate the assignments into a local array, call the function only
>> when we have enough of them).
>
>Thanks for the pointers, I will think over these points.
>
>>
>> Aside from that, I think there's a minor bug in xact.c - the patch adds
>> a "assigned" field to TransactionStateData, but then it fails to add a
>> default value into TopTransactionStateData. We probably interpret NULL
>> as false, but then there's nothing for the pointer. I suspect it might
>> leave some random garbage there, leading to strange things later.
>
>Actually, we will never access that field for the
>TopTransactionStateData, right?
>See below code, we have a check that only if IsSubTransaction(), then
>we access the "assigned" filed.
>
>+bool
>+IsSubTransactionAssignmentPending(void)
>+{
>+ if (!XLogLogicalInfoActive())
>+ return false;
>+
>+ /* we need to be in a transaction state */
>+ if (!IsTransactionState())
>+ return false;
>+
>+ /* it has to be a subtransaction */
>+ if (!IsSubTransaction())
>+ return false;
>+
>+ /* the subtransaction has to have a XID assigned */
>+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
>+ return false;
>+
>+ /* and it needs to have 'assigned' */
>+ return !CurrentTransactionState->assigned;
>+
>+}
>
The problem is not with the "assigned" field, really. AFAICS we probably
initialize it to false because we interpret NULL as false. My concern
was that we essentially leave the last pointer not initialized. That
seems like a bug, not sure if it breaks something in practice.
>>
>> Another thing I noticed is LogicalDecodingProcessRecord() extracts the
>> toplevel XID using a macro
>>
>> txid = XLogRecGetTopXid(record);
>>
>> but then it just starts accessing the fields directly again in the
>> ReorderBufferAssignChild call. I think we should do this instead:
>>
>> ReorderBufferAssignChild(ctx->reorder,
>> txid,
>> XLogRecGetXid(record),
>> buf.origptr);
>
>Make sense. I will change this in the patch.
>
+1, thanks
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-03-28 06:25 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-03-28 06:25 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Mar 4, 2020 at 9:14 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Wed, Mar 4, 2020 at 3:16 AM Tomas Vondra
> <tomas.vondra@2ndquadrant.com> wrote:
> >
> >
> > The first thing I realized that WAL-logging of assignments in v12 does
> > both the "old" logging (using dedicated message) and "new" with
> > toplevel-XID embedded in the first message. Yes, the patch was wrong,
> > because it eliminated all calls to ProcArrayApplyXidAssignment() and so
> > it was trivial to crash the replica due to KnownAssignedXids overflow.
> > But I don't think re-introducing XLOG_XACT_ASSIGNMENT message is the
> > right fix.
> >
> > I actually proposed doing this (having both ways to log assignments) so
> > that there's no regression risk with (wal_level < logical). But IIRC
> > Andres objected to it, argumenting that we should not log the same piece
> > of information in two very different ways at the same time (IIRC it was
> > discussed on the FOSDEM dev meeting, so I don't have a link to share).
> > And I do agree with him ...
> >
> > The question is, why couldn't the replica use the same assignment info
> > we already write for logical decoding? The main challenge is that now
> > the assignment can be sent in many different xlog messages, from a bunch
> > of resource managers (essentially, any xlog message with a xid can have
> > embedded XID of the toplevel xact). So the handling would either need to
> > happen in every rmgr, or we need to move it before we call the rmgr.
> >
> > For exampple, we might do this e.g. in StartupXLOG() I think, per the
> > attached patch (FWIW this particular fix was written by Masahiko Sawada,
> > not me). This does the trick for me - I'm no longer able to reproduce
> > the KnownAssignedXids overflow.
> >
> > The one difference is that we used to call ProcArrayApplyXidAssignment
> > for larger groups of XIDs, as sent in the assignment message. Now we
> > call it for each individual assignment. I don't know if this is an
> > issue, but I suppose we might introduce some sort of local caching
> > (accumulate the assignments into a local array, call the function only
> > when we have enough of them).
>
> Thanks for the pointers, I will think over these points.
>
I have looked at the solution proposed and I would like to share my
findings. I think calling ProcArrayApplyXidAssignment for each
subtransaction is not a good idea for a couple of reasons:
(a) It will just beat the purpose of maintaining KnowAssignedXids
array which is to avoid looking at pg_subtrans in
TransactionIdIsInProgress() on standby. Basically, if we remove it
for each subXid, it will consider the KnowAssignedXids to be
overflowed and check pg_subtrans frequently.
(b) Calling ProcArrayApplyXidAssignment() for each subtransaction can
be costly from the perspective of concurrency because it acquires
ProcArrayLock in Exclusive mode, so concurrently running transactions
might start blocking at this lock. Also, I see that
SubTransSetParent() makes the page dirty, so it might lead to more
writes if we spread out setting that by calling it separately for each
sub-transaction.
Apart from this, I don't see how the proposed fix is correct because
as far as I can see it tries to remove the Xid before we even record
it via RecordKnownAssignedTransactionIds(). It seems after patch
RecordKnownAssignedTransactionIds() will be called after
ProcArrayApplyXidAssignment(), how could that be correct.
Thoughts?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-03-28 08:49 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-03-28 08:49 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Mar 28, 2020 at 11:56 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, Mar 4, 2020 at 9:14 AM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Wed, Mar 4, 2020 at 3:16 AM Tomas Vondra
> > <tomas.vondra@2ndquadrant.com> wrote:
> > >
> > >
> > > The first thing I realized that WAL-logging of assignments in v12 does
> > > both the "old" logging (using dedicated message) and "new" with
> > > toplevel-XID embedded in the first message. Yes, the patch was wrong,
> > > because it eliminated all calls to ProcArrayApplyXidAssignment() and so
> > > it was trivial to crash the replica due to KnownAssignedXids overflow.
> > > But I don't think re-introducing XLOG_XACT_ASSIGNMENT message is the
> > > right fix.
> > >
> > > I actually proposed doing this (having both ways to log assignments) so
> > > that there's no regression risk with (wal_level < logical). But IIRC
> > > Andres objected to it, argumenting that we should not log the same piece
> > > of information in two very different ways at the same time (IIRC it was
> > > discussed on the FOSDEM dev meeting, so I don't have a link to share).
> > > And I do agree with him ...
> > >
> > > The question is, why couldn't the replica use the same assignment info
> > > we already write for logical decoding? The main challenge is that now
> > > the assignment can be sent in many different xlog messages, from a bunch
> > > of resource managers (essentially, any xlog message with a xid can have
> > > embedded XID of the toplevel xact). So the handling would either need to
> > > happen in every rmgr, or we need to move it before we call the rmgr.
> > >
> > > For exampple, we might do this e.g. in StartupXLOG() I think, per the
> > > attached patch (FWIW this particular fix was written by Masahiko Sawada,
> > > not me). This does the trick for me - I'm no longer able to reproduce
> > > the KnownAssignedXids overflow.
> > >
> > > The one difference is that we used to call ProcArrayApplyXidAssignment
> > > for larger groups of XIDs, as sent in the assignment message. Now we
> > > call it for each individual assignment. I don't know if this is an
> > > issue, but I suppose we might introduce some sort of local caching
> > > (accumulate the assignments into a local array, call the function only
> > > when we have enough of them).
> >
> > Thanks for the pointers, I will think over these points.
> >
>
> I have looked at the solution proposed and I would like to share my
> findings. I think calling ProcArrayApplyXidAssignment for each
> subtransaction is not a good idea for a couple of reasons:
> (a) It will just beat the purpose of maintaining KnowAssignedXids
> array which is to avoid looking at pg_subtrans in
> TransactionIdIsInProgress() on standby. Basically, if we remove it
> for each subXid, it will consider the KnowAssignedXids to be
> overflowed and check pg_subtrans frequently.
Right, I also think this is a problem with this solution. I think we
may try to avoid this by caching this information. But, then we will
have to maintain this in some dimensional array which stores
sub-transaction ids per top transaction or we can maintain a list of
sub-transaction for each transaction. I haven't thought about how
much complexity this solution will add.
> (b) Calling ProcArrayApplyXidAssignment() for each subtransaction can
> be costly from the perspective of concurrency because it acquires
> ProcArrayLock in Exclusive mode, so concurrently running transactions
> might start blocking at this lock.
Right
Also, I see that
> SubTransSetParent() makes the page dirty, so it might lead to more
> writes if we spread out setting that by calling it separately for each
> sub-transaction.
Right.
>
> Apart from this, I don't see how the proposed fix is correct because
> as far as I can see it tries to remove the Xid before we even record
> it via RecordKnownAssignedTransactionIds(). It seems after patch
> RecordKnownAssignedTransactionIds() will be called after
> ProcArrayApplyXidAssignment(), how could that be correct.
Valid point.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-03-28 09:59 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-03-28 09:59 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Mar 28, 2020 at 2:19 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sat, Mar 28, 2020 at 11:56 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> >
> > I have looked at the solution proposed and I would like to share my
> > findings. I think calling ProcArrayApplyXidAssignment for each
> > subtransaction is not a good idea for a couple of reasons:
> > (a) It will just beat the purpose of maintaining KnowAssignedXids
> > array which is to avoid looking at pg_subtrans in
> > TransactionIdIsInProgress() on standby. Basically, if we remove it
> > for each subXid, it will consider the KnowAssignedXids to be
> > overflowed and check pg_subtrans frequently.
>
> Right, I also think this is a problem with this solution. I think we
> may try to avoid this by caching this information. But, then we will
> have to maintain this in some dimensional array which stores
> sub-transaction ids per top transaction or we can maintain a list of
> sub-transaction for each transaction. I haven't thought about how
> much complexity this solution will add.
>
How about if instead of writing an XLOG_XACT_ASSIGNMENT WAL, we set a
flag in TransactionStateData and then log that as special information
whenever we write next WAL record for a new subtransaction? Then
during recovery, we can only call ProcArrayApplyXidAssignment when we
find that special flag is set in a WAL record. One idea could be to
use a flag bit in XLogRecord.xl_info. If that is feasible then the
solution can work as it is now, without any overhead or change in the
way we maintain KnownAssignedXids.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-03-29 00:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2020-03-29 00:59 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Mar 28, 2020 at 03:29:34PM +0530, Amit Kapila wrote:
>On Sat, Mar 28, 2020 at 2:19 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>>
>> On Sat, Mar 28, 2020 at 11:56 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >
>> >
>> > I have looked at the solution proposed and I would like to share my
>> > findings. I think calling ProcArrayApplyXidAssignment for each
>> > subtransaction is not a good idea for a couple of reasons:
>> > (a) It will just beat the purpose of maintaining KnowAssignedXids
>> > array which is to avoid looking at pg_subtrans in
>> > TransactionIdIsInProgress() on standby. Basically, if we remove it
>> > for each subXid, it will consider the KnowAssignedXids to be
>> > overflowed and check pg_subtrans frequently.
>>
>> Right, I also think this is a problem with this solution. I think we
>> may try to avoid this by caching this information. But, then we will
>> have to maintain this in some dimensional array which stores
>> sub-transaction ids per top transaction or we can maintain a list of
>> sub-transaction for each transaction. I haven't thought about how
>> much complexity this solution will add.
>>
>
>How about if instead of writing an XLOG_XACT_ASSIGNMENT WAL, we set a
>flag in TransactionStateData and then log that as special information
>whenever we write next WAL record for a new subtransaction? Then
>during recovery, we can only call ProcArrayApplyXidAssignment when we
>find that special flag is set in a WAL record. One idea could be to
>use a flag bit in XLogRecord.xl_info. If that is feasible then the
>solution can work as it is now, without any overhead or change in the
>way we maintain KnownAssignedXids.
>
Ummm, how is that different from what the patch is doing now? I mean, we
only write the top-level XID for the first WAL record in each subxact,
right? Or what would be the difference with your approach?
Anyway, I think you're right the ProcArrayApplyXidAssignment call was
done too early, but I think that can be fixed by moving it until after
the RecordKnownAssignedTransactionIds call, no? Essentially, right
before rm_redo().
You're right calling ProcArrayApplyXidAssignment() may be an issue,
because it exclusively acquires the ProcArrayLock. I've actually hinted
that might be an issue in my original message, suggesting we might add a
local cache of assigned XIDs (a small static array, doing essentially
the same thing we used to do on the upstream node). I haven't done that
in my WIP patch to keep it simple, but AFACS it'd work.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2020-03-29 05:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-03-29 05:49 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Mar 29, 2020 at 6:29 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Sat, Mar 28, 2020 at 03:29:34PM +0530, Amit Kapila wrote:
> >On Sat, Mar 28, 2020 at 2:19 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> >How about if instead of writing an XLOG_XACT_ASSIGNMENT WAL, we set a
> >flag in TransactionStateData and then log that as special information
> >whenever we write next WAL record for a new subtransaction? Then
> >during recovery, we can only call ProcArrayApplyXidAssignment when we
> >find that special flag is set in a WAL record. One idea could be to
> >use a flag bit in XLogRecord.xl_info. If that is feasible then the
> >solution can work as it is now, without any overhead or change in the
> >way we maintain KnownAssignedXids.
> >
>
> Ummm, how is that different from what the patch is doing now? I mean, we
> only write the top-level XID for the first WAL record in each subxact,
> right? Or what would be the difference with your approach?
>
We have to do what the patch is currently doing and additionally, we
will set this flag after PGPROC_MAX_CACHED_SUBXIDS which would allow
us to call ProcArrayApplyXidAssignment during WAL replay only after
PGPROC_MAX_CACHED_SUBXIDS number of subxacts. It will help us in
clearing the KnownAssignedXids at the same time as we do now, so no
additional performance overhead.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-03-29 15:31 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2020-03-29 15:31 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Mar 29, 2020 at 11:19:21AM +0530, Amit Kapila wrote:
>On Sun, Mar 29, 2020 at 6:29 AM Tomas Vondra
><tomas.vondra@2ndquadrant.com> wrote:
>>
>> On Sat, Mar 28, 2020 at 03:29:34PM +0530, Amit Kapila wrote:
>> >On Sat, Mar 28, 2020 at 2:19 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>> >
>> >How about if instead of writing an XLOG_XACT_ASSIGNMENT WAL, we set a
>> >flag in TransactionStateData and then log that as special information
>> >whenever we write next WAL record for a new subtransaction? Then
>> >during recovery, we can only call ProcArrayApplyXidAssignment when we
>> >find that special flag is set in a WAL record. One idea could be to
>> >use a flag bit in XLogRecord.xl_info. If that is feasible then the
>> >solution can work as it is now, without any overhead or change in the
>> >way we maintain KnownAssignedXids.
>> >
>>
>> Ummm, how is that different from what the patch is doing now? I mean, we
>> only write the top-level XID for the first WAL record in each subxact,
>> right? Or what would be the difference with your approach?
>>
>
>We have to do what the patch is currently doing and additionally, we
>will set this flag after PGPROC_MAX_CACHED_SUBXIDS which would allow
>us to call ProcArrayApplyXidAssignment during WAL replay only after
>PGPROC_MAX_CACHED_SUBXIDS number of subxacts. It will help us in
>clearing the KnownAssignedXids at the same time as we do now, so no
>additional performance overhead.
>
Hmmm. So we'd still log assignment twice? Or would we keep just the
immediate assignments (embedded into xlog records), and cache the
subxids on the replica somehow?
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2020-03-30 06:17 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-03-30 06:17 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sun, Mar 29, 2020 at 9:01 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Sun, Mar 29, 2020 at 11:19:21AM +0530, Amit Kapila wrote:
> >On Sun, Mar 29, 2020 at 6:29 AM Tomas Vondra
> ><tomas.vondra@2ndquadrant.com> wrote:
> >>
> >> Ummm, how is that different from what the patch is doing now? I mean, we
> >> only write the top-level XID for the first WAL record in each subxact,
> >> right? Or what would be the difference with your approach?
> >>
> >
> >We have to do what the patch is currently doing and additionally, we
> >will set this flag after PGPROC_MAX_CACHED_SUBXIDS which would allow
> >us to call ProcArrayApplyXidAssignment during WAL replay only after
> >PGPROC_MAX_CACHED_SUBXIDS number of subxacts. It will help us in
> >clearing the KnownAssignedXids at the same time as we do now, so no
> >additional performance overhead.
> >
>
> Hmmm. So we'd still log assignment twice? Or would we keep just the
> immediate assignments (embedded into xlog records), and cache the
> subxids on the replica somehow?
>
I think we need to cache the subxids on the replica somehow but I
don't have a very good idea for it. Basically, there are two ways to
do it (a) Change the KnownAssignedXids in some way so that we can
easily find this information without losing on the current benefits of
it. I can't think of a good way to do that and even if we come up
with something, it could easily be a lot of work, (b) Cache the
subxids for a particular transaction in local memory along with
KnownAssignedXids. This is doable but now we have two data-structures
(one in shared memory and other in local memory) managing the same
information in different ways.
Do you have any other ideas?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-03-30 15:27 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Tomas Vondra @ 2020-03-30 15:27 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Mar 30, 2020 at 11:47:57AM +0530, Amit Kapila wrote:
>On Sun, Mar 29, 2020 at 9:01 PM Tomas Vondra
><tomas.vondra@2ndquadrant.com> wrote:
>>
>> On Sun, Mar 29, 2020 at 11:19:21AM +0530, Amit Kapila wrote:
>> >On Sun, Mar 29, 2020 at 6:29 AM Tomas Vondra
>> ><tomas.vondra@2ndquadrant.com> wrote:
>> >>
>> >> Ummm, how is that different from what the patch is doing now? I mean, we
>> >> only write the top-level XID for the first WAL record in each subxact,
>> >> right? Or what would be the difference with your approach?
>> >>
>> >
>> >We have to do what the patch is currently doing and additionally, we
>> >will set this flag after PGPROC_MAX_CACHED_SUBXIDS which would allow
>> >us to call ProcArrayApplyXidAssignment during WAL replay only after
>> >PGPROC_MAX_CACHED_SUBXIDS number of subxacts. It will help us in
>> >clearing the KnownAssignedXids at the same time as we do now, so no
>> >additional performance overhead.
>> >
>>
>> Hmmm. So we'd still log assignment twice? Or would we keep just the
>> immediate assignments (embedded into xlog records), and cache the
>> subxids on the replica somehow?
>>
>
>I think we need to cache the subxids on the replica somehow but I
>don't have a very good idea for it. Basically, there are two ways to
>do it (a) Change the KnownAssignedXids in some way so that we can
>easily find this information without losing on the current benefits of
>it. I can't think of a good way to do that and even if we come up
>with something, it could easily be a lot of work, (b) Cache the
>subxids for a particular transaction in local memory along with
>KnownAssignedXids. This is doable but now we have two data-structures
>(one in shared memory and other in local memory) managing the same
>information in different ways.
>
>Do you have any other ideas?
I don't follow. Why couldn't we have a simple cache on the standby? It
could be either a simple array or a hash table (with the top-level xid
as hash key)?
I think the single array would be sufficient, but the hash table would
allow keeping the apply logic more or less as it's today. See the
attached patch that adds such cache - I do admit I haven't tested this,
but hopefully it's a sufficient illustration of the idea.
It does not handle cleanup of the cache, but I think that should not be
difficult - we simply need to remove entries for transactions that got
committed or rolled back. And do something about transactions without an
explicit commit/rollback record, but that can be done by also handling
XLOG_RUNNING_XACTS (by removing anything preceding oldestRunningXid).
I don't think this is particularly complicated or a lot of code, and I
don't see why would it require data structures in shared memory. Only
the walreceiver on standby needs to worry about this, no?
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 1951103b26..b85c046b41 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7300,6 +7300,19 @@ StartupXLOG(void)
TransactionIdIsValid(record->xl_xid))
RecordKnownAssignedTransactionIds(record->xl_xid);
+ /*
+ * Assign subtransaction xids to the top level xid if the
+ * record has that information. This is required at most
+ * once per subtransactions.
+ */
+ if (TransactionIdIsValid(xlogreader->toplevel_xid) &&
+ standbyState >= STANDBY_INITIALIZED)
+ {
+ Assert(XLogStandbyInfoActive());
+ ProcArrayCacheXidAssignment(xlogreader->toplevel_xid,
+ record->xl_xid);
+ }
+
/* Now apply the WAL record itself */
RmgrTable[record->xl_rmid].rm_redo(xlogreader);
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cfb88db4a4..efddd0e1e6 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -958,6 +958,53 @@ ProcArrayApplyXidAssignment(TransactionId topxid,
LWLockRelease(ProcArrayLock);
}
+static HTAB *xidAssignmentsHash = NULL;
+
+typedef struct XidAssignmentEntry
+{
+ /* the hash lookup key MUST BE FIRST */
+ TransactionId topXid;
+
+ int nsubxids;
+ TransactionId subxids[PGPROC_MAX_CACHED_SUBXIDS];
+} XidAssignmentEntry;
+
+void
+ProcArrayCacheXidAssignment(TransactionId topXid, TransactionId subXid)
+{
+ XidAssignmentEntry *entry;
+ bool found;
+
+ if (xidAssignmentsHash == NULL)
+ {
+ /* First time through: initialize the hash table */
+ HASHCTL ctl;
+
+ MemSet(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(TransactionId);
+ ctl.entrysize = sizeof(XidAssignmentEntry);
+ xidAssignmentsHash = hash_create("XID assignment cache", 256,
+ &ctl, HASH_ELEM | HASH_BLOBS);
+ }
+
+ /* Look for an existing entry */
+ entry = (XidAssignmentEntry *) hash_search(xidAssignmentsHash,
+ (void *) &topXid,
+ HASH_ENTER, &found);
+
+ if (!found)
+ entry->nsubxids = 0;
+
+ entry->subxids[entry->nsubxids++] = subXid;
+
+ /* after reaching the limit, apply the assignments for this top XID */
+ if (entry->nsubxids == PGPROC_MAX_CACHED_SUBXIDS)
+ {
+ ProcArrayApplyXidAssignment(topXid, entry->nsubxids, entry->subxids);
+ entry->nsubxids = 0;
+ }
+}
+
/*
* TransactionIdIsInProgress -- is given transaction running in some backend
*
diff --git a/src/include/storage/procarray.h b/src/include/storage/procarray.h
index a5c7d0c064..17d89bdd7e 100644
--- a/src/include/storage/procarray.h
+++ b/src/include/storage/procarray.h
@@ -67,6 +67,7 @@ extern void ProcArrayInitRecovery(TransactionId initializedUptoXID);
extern void ProcArrayApplyRecoveryInfo(RunningTransactions running);
extern void ProcArrayApplyXidAssignment(TransactionId topxid,
int nsubxids, TransactionId *subxids);
+extern void ProcArrayCacheXidAssignment(TransactionId topxid, TransactionId subxid);
extern void RecordKnownAssignedTransactionIds(TransactionId xid);
extern void ExpireTreeKnownAssignedTransactionIds(TransactionId xid,
Attachments:
xid-assignment-v13-fix.patch (text/plain, 3.1K) download
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 1951103b26..b85c046b41 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -7300,6 +7300,19 @@ StartupXLOG(void)
TransactionIdIsValid(record->xl_xid))
RecordKnownAssignedTransactionIds(record->xl_xid);
+ /*
+ * Assign subtransaction xids to the top level xid if the
+ * record has that information. This is required at most
+ * once per subtransactions.
+ */
+ if (TransactionIdIsValid(xlogreader->toplevel_xid) &&
+ standbyState >= STANDBY_INITIALIZED)
+ {
+ Assert(XLogStandbyInfoActive());
+ ProcArrayCacheXidAssignment(xlogreader->toplevel_xid,
+ record->xl_xid);
+ }
+
/* Now apply the WAL record itself */
RmgrTable[record->xl_rmid].rm_redo(xlogreader);
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index cfb88db4a4..efddd0e1e6 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -958,6 +958,53 @@ ProcArrayApplyXidAssignment(TransactionId topxid,
LWLockRelease(ProcArrayLock);
}
+static HTAB *xidAssignmentsHash = NULL;
+
+typedef struct XidAssignmentEntry
+{
+ /* the hash lookup key MUST BE FIRST */
+ TransactionId topXid;
+
+ int nsubxids;
+ TransactionId subxids[PGPROC_MAX_CACHED_SUBXIDS];
+} XidAssignmentEntry;
+
+void
+ProcArrayCacheXidAssignment(TransactionId topXid, TransactionId subXid)
+{
+ XidAssignmentEntry *entry;
+ bool found;
+
+ if (xidAssignmentsHash == NULL)
+ {
+ /* First time through: initialize the hash table */
+ HASHCTL ctl;
+
+ MemSet(&ctl, 0, sizeof(ctl));
+ ctl.keysize = sizeof(TransactionId);
+ ctl.entrysize = sizeof(XidAssignmentEntry);
+ xidAssignmentsHash = hash_create("XID assignment cache", 256,
+ &ctl, HASH_ELEM | HASH_BLOBS);
+ }
+
+ /* Look for an existing entry */
+ entry = (XidAssignmentEntry *) hash_search(xidAssignmentsHash,
+ (void *) &topXid,
+ HASH_ENTER, &found);
+
+ if (!found)
+ entry->nsubxids = 0;
+
+ entry->subxids[entry->nsubxids++] = subXid;
+
+ /* after reaching the limit, apply the assignments for this top XID */
+ if (entry->nsubxids == PGPROC_MAX_CACHED_SUBXIDS)
+ {
+ ProcArrayApplyXidAssignment(topXid, entry->nsubxids, entry->subxids);
+ entry->nsubxids = 0;
+ }
+}
+
/*
* TransactionIdIsInProgress -- is given transaction running in some backend
*
diff --git a/src/include/storage/procarray.h b/src/include/storage/procarray.h
index a5c7d0c064..17d89bdd7e 100644
--- a/src/include/storage/procarray.h
+++ b/src/include/storage/procarray.h
@@ -67,6 +67,7 @@ extern void ProcArrayInitRecovery(TransactionId initializedUptoXID);
extern void ProcArrayApplyRecoveryInfo(RunningTransactions running);
extern void ProcArrayApplyXidAssignment(TransactionId topxid,
int nsubxids, TransactionId *subxids);
+extern void ProcArrayCacheXidAssignment(TransactionId topxid, TransactionId subxid);
extern void RecordKnownAssignedTransactionIds(TransactionId xid);
extern void ExpireTreeKnownAssignedTransactionIds(TransactionId xid,
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2020-04-07 06:47 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-04-07 06:47 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Mar 30, 2020 at 8:58 PM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Mon, Mar 30, 2020 at 11:47:57AM +0530, Amit Kapila wrote:
> >
> >I think we need to cache the subxids on the replica somehow but I
> >don't have a very good idea for it. Basically, there are two ways to
> >do it (a) Change the KnownAssignedXids in some way so that we can
> >easily find this information without losing on the current benefits of
> >it. I can't think of a good way to do that and even if we come up
> >with something, it could easily be a lot of work, (b) Cache the
> >subxids for a particular transaction in local memory along with
> >KnownAssignedXids. This is doable but now we have two data-structures
> >(one in shared memory and other in local memory) managing the same
> >information in different ways.
> >
> >Do you have any other ideas?
>
> I don't follow. Why couldn't we have a simple cache on the standby? It
> could be either a simple array or a hash table (with the top-level xid
> as hash key)?
>
I think having something like we discussed or what you have in the
patch won't be sufficient to clean the KnownAssignedXid array. The
point is that we won't write a WAL for xid-subxid association for
unlogged relations in the "Immediately WAL-log assignments" patch,
however, the KnownAssignedXid would have both kinds of Xids as we
autofill it with gaps (see RecordKnownAssignedTransactionIds). I
think if my understanding is correct to make it work we might need
major surgery in the code or have to maintain KnownAssignedXid array
differently.
>
> I don't think this is particularly complicated or a lot of code, and I
> don't see why would it require data structures in shared memory. Only
> the walreceiver on standby needs to worry about this, no?
>
Not a new data structure in shared memory, but we already have a
KnownTransactionId structure in shared memory. So, after having a
local cache, we will have xidAssignmentsHash and KnownTransactionId
maintaining the same information in different ways. And, we need to
ensure both are cleaned up properly. That was what I was pointing
above related to maintaining two structures. However, I think before
discussing more on this, we need to think about the above problem.
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-04-08 00:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 06:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Tomas Vondra @ 2020-04-08 00:59 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Dilip Kumar <dilipbalaut@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Apr 07, 2020 at 12:17:44PM +0530, Amit Kapila wrote:
>On Mon, Mar 30, 2020 at 8:58 PM Tomas Vondra
><tomas.vondra@2ndquadrant.com> wrote:
>>
>> On Mon, Mar 30, 2020 at 11:47:57AM +0530, Amit Kapila wrote:
>> >
>> >I think we need to cache the subxids on the replica somehow but I
>> >don't have a very good idea for it. Basically, there are two ways to
>> >do it (a) Change the KnownAssignedXids in some way so that we can
>> >easily find this information without losing on the current benefits of
>> >it. I can't think of a good way to do that and even if we come up
>> >with something, it could easily be a lot of work, (b) Cache the
>> >subxids for a particular transaction in local memory along with
>> >KnownAssignedXids. This is doable but now we have two data-structures
>> >(one in shared memory and other in local memory) managing the same
>> >information in different ways.
>> >
>> >Do you have any other ideas?
>>
>> I don't follow. Why couldn't we have a simple cache on the standby? It
>> could be either a simple array or a hash table (with the top-level xid
>> as hash key)?
>>
>
>I think having something like we discussed or what you have in the
>patch won't be sufficient to clean the KnownAssignedXid array. The
>point is that we won't write a WAL for xid-subxid association for
>unlogged relations in the "Immediately WAL-log assignments" patch,
>however, the KnownAssignedXid would have both kinds of Xids as we
>autofill it with gaps (see RecordKnownAssignedTransactionIds). I
>think if my understanding is correct to make it work we might need
>major surgery in the code or have to maintain KnownAssignedXid array
>differently.
Hmm, that's a good point. If I understand correctly, the issue is
that if we create new subxact, write something into an unlogged table,
and then create new subxact, the XID of the first subxact will be "known
assigned" but we won't know it's a subxact or to which parent xact it
belongs (because there will be no WAL records that could encode it).
I wonder if there's a simple solution (e.g. when creating the second
subxact we might notice the xid-subxid assignment was not logged, and
write some "dummy" WAL record). But I admit it seems a bit ugly.
>>
>> I don't think this is particularly complicated or a lot of code, and I
>> don't see why would it require data structures in shared memory. Only
>> the walreceiver on standby needs to worry about this, no?
>>
>
>Not a new data structure in shared memory, but we already have a
>KnownTransactionId structure in shared memory. So, after having a
>local cache, we will have xidAssignmentsHash and KnownTransactionId
>maintaining the same information in different ways. And, we need to
>ensure both are cleaned up properly. That was what I was pointing
>above related to maintaining two structures. However, I think before
>discussing more on this, we need to think about the above problem.
>
Sure.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
@ 2020-04-09 09:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-04-09 09:10 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Apr 8, 2020 at 6:29 AM Tomas Vondra
<tomas.vondra@2ndquadrant.com> wrote:
>
> On Tue, Apr 07, 2020 at 12:17:44PM +0530, Amit Kapila wrote:
> >On Mon, Mar 30, 2020 at 8:58 PM Tomas Vondra
> ><tomas.vondra@2ndquadrant.com> wrote:
> >>
> >> On Mon, Mar 30, 2020 at 11:47:57AM +0530, Amit Kapila wrote:
> >> >
> >> >I think we need to cache the subxids on the replica somehow but I
> >> >don't have a very good idea for it. Basically, there are two ways to
> >> >do it (a) Change the KnownAssignedXids in some way so that we can
> >> >easily find this information without losing on the current benefits of
> >> >it. I can't think of a good way to do that and even if we come up
> >> >with something, it could easily be a lot of work, (b) Cache the
> >> >subxids for a particular transaction in local memory along with
> >> >KnownAssignedXids. This is doable but now we have two data-structures
> >> >(one in shared memory and other in local memory) managing the same
> >> >information in different ways.
> >> >
> >> >Do you have any other ideas?
> >>
> >> I don't follow. Why couldn't we have a simple cache on the standby? It
> >> could be either a simple array or a hash table (with the top-level xid
> >> as hash key)?
> >>
> >
> >I think having something like we discussed or what you have in the
> >patch won't be sufficient to clean the KnownAssignedXid array. The
> >point is that we won't write a WAL for xid-subxid association for
> >unlogged relations in the "Immediately WAL-log assignments" patch,
> >however, the KnownAssignedXid would have both kinds of Xids as we
> >autofill it with gaps (see RecordKnownAssignedTransactionIds). I
> >think if my understanding is correct to make it work we might need
> >major surgery in the code or have to maintain KnownAssignedXid array
> >differently.
>
> Hmm, that's a good point. If I understand correctly, the issue is
> that if we create new subxact, write something into an unlogged table,
> and then create new subxact, the XID of the first subxact will be "known
> assigned" but we won't know it's a subxact or to which parent xact it
> belongs (because there will be no WAL records that could encode it).
>
> I wonder if there's a simple solution (e.g. when creating the second
> subxact we might notice the xid-subxid assignment was not logged, and
> write some "dummy" WAL record). But I admit it seems a bit ugly.
>
> >>
> >> I don't think this is particularly complicated or a lot of code, and I
> >> don't see why would it require data structures in shared memory. Only
> >> the walreceiver on standby needs to worry about this, no?
> >>
> >
> >Not a new data structure in shared memory, but we already have a
> >KnownTransactionId structure in shared memory. So, after having a
> >local cache, we will have xidAssignmentsHash and KnownTransactionId
> >maintaining the same information in different ways. And, we need to
> >ensure both are cleaned up properly. That was what I was pointing
> >above related to maintaining two structures. However, I think before
> >discussing more on this, we need to think about the above problem.
I have rebased the patch on the latest head. I haven't yet changed
anything for xid assignment thing because it is not yet concluded.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v13-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.5K) download
From 314a9919eff3f7399a2ba05f3d9ae192854e7d0d Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 20 Mar 2020 15:03:01 +0530
Subject: [PATCH v13 01/10] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 46 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 +++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 ++++++++++++++-------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 99 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 6b1ae1f..c5842d3 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -222,6 +223,7 @@ typedef struct SerializedTransactionState
static TransactionStateData TopTransactionStateData = {
.state = TRANS_DEFAULT,
.blockState = TBLOCK_DEFAULT,
+ .assigned = false,
};
/*
@@ -5111,6 +5113,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6013,3 +6016,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 4259309..afae08d 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -89,11 +89,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -195,6 +197,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -398,7 +404,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -748,6 +754,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 79ff976..7b5257f 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1189,6 +1189,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1227,6 +1228,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index c2e5e3a..0946179 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ txid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrId) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index f60ed2d..6d439d0 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -229,6 +229,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 4582196..756f6df 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -150,6 +150,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -285,6 +287,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v13-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch (application/octet-stream, 12.6K) download
From 6ea334436c9caa80ea1534780c5d498723ebadbb Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Apr 2020 10:55:19 +0530
Subject: [PATCH v13 04/10] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 40 ++++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++++--
src/include/utils/snapmgr.h | 4 ++-
6 files changed, 115 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 65244b1..b59a6c8 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>pg_current_xact_id()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index c4a5aa6..c1586ef 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1431,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1536,6 +1553,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1685,6 +1710,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5490,6 +5523,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index dfba5ae..413a21f 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -481,6 +482,19 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
@@ -517,6 +531,19 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return result;
}
@@ -643,6 +670,19 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we
+ * error out. We can't directly use TransactionIdDidAbort as after crash
+ * such transaction might not have been marked as aborted. See detailed
+ * comments at snapmgr.c where the variable is declared.
+ */
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 183a0e9..e299142 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -696,7 +696,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1547,7 +1547,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1798,7 +1798,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1818,7 +1818,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v13-0002-Issue-individual-invalidations-with-wal_level-lo.patch (application/octet-stream, 17.9K) download
From 97b37f9390442504dba085ce896ca5a0aad6b61b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Mon, 18 Nov 2019 16:26:33 +0530
Subject: [PATCH v13 02/10] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
The new invalidations are written to WAL immediately, without any
such caching. Perhaps it would be possible to add similar caching,
e.g. at the command level, or something like that?
---
src/backend/access/rmgrdesc/xactdesc.c | 40 +++++++++
src/backend/access/transam/xact.c | 7 ++
src/backend/replication/logical/decode.c | 18 ++++
src/backend/replication/logical/reorderbuffer.c | 110 +++++++++++++++++++++---
src/backend/utils/cache/inval.c | 59 +++++++++++--
src/include/access/xact.h | 13 ++-
src/include/replication/reorderbuffer.h | 11 +++
7 files changed, 239 insertions(+), 19 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index fbc5942..17c06f7 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,9 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -396,6 +399,12 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs);
+ }
}
const char *
@@ -423,7 +432,38 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs)
+{
+ int i;
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index c5842d3..cf78ffc 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6013,6 +6013,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 0946179..4f958e9 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,24 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->nmsgs, invals->msgs);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 4594cf9..183a0e9 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -220,7 +220,7 @@ static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
-static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs);
/*
* ---------------------------------------
@@ -455,6 +455,11 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
pfree(change->data.msg.message);
change->data.msg.message = NULL;
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ if (change->data.inval.invalidations)
+ pfree(change->data.inval.invalidations);
+ change->data.inval.invalidations = NULL;
+ break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
if (change->data.snapshot)
{
@@ -1814,17 +1819,24 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation messages locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ ReorderBufferExecuteInvalidations(
+ change->data.inval.ninvalidations,
+ change->data.inval.invalidations);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -1866,7 +1878,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -1892,7 +1905,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2204,6 +2218,39 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, int nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ MemoryContextAlloc(rb->context,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2234,12 +2281,12 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
* in the changestream but we don't know which those are.
*/
static void
-ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs)
{
int i;
- for (i = 0; i < txn->ninvalidations; i++)
- LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+ for (i = 0; i < nmsgs; i++)
+ LocalExecuteInvalidationMessage(&msgs[i]);
}
/*
@@ -2591,6 +2638,24 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ {
+ char *data;
+ Size inval_size = sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+
+ sz += inval_size;
+
+ ReorderBufferSerializeReserve(rb, sz);
+ data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
+
+ /* might have been reallocated above */
+ ondisk = (ReorderBufferDiskChange *) rb->outbuf;
+ memcpy(data, change->data.inval.invalidations, inval_size);
+ data += inval_size;
+
+ break;
+ }
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot snap;
@@ -2736,6 +2801,11 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ sz += sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot snap;
@@ -3004,6 +3074,20 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ {
+ Size inval_size = sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+
+ change->data.inval.invalidations =
+ MemoryContextAlloc(rb->context,
+ change->data.msg.message_size);
+ /* read the message */
+ memcpy(change->data.inval.invalidations, data, inval_size);
+ data += inval_size;
+
+ break;
+ }
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot oldsnap;
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..47be680 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write invalidations into WAL at each command end to
+ * support the decoding of the in-progress transaction. As of now it was
+ * enough to log invalidation only at commit because we are only decoding the
+ * transaction at the commit time. We only need to log the catalog cache and
+ * relcache invalidation. There can not be any active MVCC scan in logical
+ * decoding so we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,8 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(void);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -1068,8 +1077,8 @@ AtEOSubXact_Inval(bool isCommit)
/*
* CommandEndInvalidationMessages
- * Process queued-up invalidation messages at end of one command
- * in a transaction.
+ * Process queued-up invalidation messages at end of one command
+ * in a transaction.
*
* Here, we send no messages to the shared queue, since we don't know yet if
* we will commit. We do need to locally process the CurrentCmdInvalidMsgs
@@ -1078,8 +1087,8 @@ AtEOSubXact_Inval(bool isCommit)
* of the prior-cmds list.
*
* Note:
- * This should be called during CommandCounterIncrement(),
- * after we have advanced the command ID.
+ * This should be called during CommandCounterIncrement(),
+ * after we have advanced the command ID.
*/
void
CommandEndInvalidationMessages(void)
@@ -1090,7 +1099,10 @@ CommandEndInvalidationMessages(void)
* just quietly return if no state to work on.
*/
if (transInvalInfo == NULL)
- return;
+ return;
+
+ if (XLogLogicalInfoActive())
+ LogLogicalInvalidations();
ProcessInvalidationMessages(&transInvalInfo->CurrentCmdInvalidMsgs,
LocalExecuteInvalidationMessage);
@@ -1501,3 +1513,40 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations()
+{
+ xl_xact_invalidations xlrec;
+ SharedInvalidationMessage *invalMessages;
+ int nmsgs = 0;
+
+ if (transInvalInfo->CurrentCmdInvalidMsgs.cclist)
+ {
+ ProcessInvalidationMessagesMulti(&transInvalInfo->CurrentCmdInvalidMsgs,
+ MakeSharedInvalidMessagesArray);
+ invalMessages = SharedInvalidMessagesArray;
+ nmsgs = numSharedInvalidMessagesArray;
+ SharedInvalidMessagesArray = NULL;
+ numSharedInvalidMessagesArray = 0;
+ }
+
+ if (nmsgs > 0)
+ {
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) invalMessages,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+
+ pfree(invalMessages);
+ }
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..b822c5e 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,17 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ */
+typedef struct xl_xact_invalidations
+{
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 626ecf4..af35287 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,14 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ uint32 ninvalidations; /* Number of messages */
+ SharedInvalidationMessage *invalidations; /* invalidation
+ * message */
+ } inval;
} data;
/*
@@ -459,6 +468,8 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ int nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v13-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From 07523af1992bc0a53ed25f176eff1add2571b9f4 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v13 03/10] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 93c9488..64f651f 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -540,3 +569,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index bad3bfe..65244b1 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 5adf253..497d8a9 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -862,6 +910,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index af35287..e102840 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -354,6 +354,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -393,6 +439,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v13-0005-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 37.8K) download
From b44551942dd95833421fe85b4225def6da48bc2b Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v13 05/10] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 710 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 36 ++
3 files changed, 693 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index e299142..7383f14 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -773,6 +786,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -865,6 +910,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -988,7 +1036,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1024,6 +1072,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1038,6 +1089,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1316,6 +1370,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1341,8 +1404,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1350,9 +1498,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1491,63 +1636,75 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+/*
+ * Set xid for concurrent abort check
+ *
+ * While streaming an in-progress transaction there is a possibility that the
+ * (sub)transaction might get aborted concurrently. In such case if the
+ * (sub)transaction has catalog update then we might decode the tuple using
+ * wrong catalog version. So for detecting the concurrent abort we set
+ * CheckXidAlive to the current (sub)transaction's xid for which this change
+ * belongs to. And, during catalog scan we can check the status of the xid and
+ * if it is aborted we will report an specific error which we can ignore. We
+ * might have already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the abort we will
+ * stream abort message to truncate the changes in the subscriber.
+ */
+static inline void
+SetupCheckXidLive(TransactionId xid)
+{
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
+ if (!TransactionIdDidCommit(xid))
+ CheckXidAlive = xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
- snapshot_now = txn->base_snapshot;
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
- /* build data to be able to lookup the CommandIds of catalog tuples */
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1563,15 +1720,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1579,6 +1741,19 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* Set the xid for concurrent abort check. */
+ if (streaming)
+ SetupCheckXidLive(change->txn->xid);
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1588,8 +1763,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1655,7 +1828,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1676,8 +1857,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1695,7 +1874,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1753,7 +1932,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1762,10 +1949,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1796,9 +1989,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1818,7 +2011,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1858,14 +2052,46 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ if (!XLogRecPtrIsInvalid(prev_lsn))
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1884,14 +2110,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,18 +2145,122 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1946,6 +2284,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2015,6 +2360,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2150,8 +2502,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2159,6 +2520,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2170,19 +2532,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2211,6 +2582,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2301,6 +2673,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2405,6 +2784,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2424,15 +2835,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- ReorderBufferSerializeTXN(rb, txn);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2752,6 +3194,102 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after the last
+ * streaming run, we might have got some new sub-transactions. So we
+ * need to add them to the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ Assert(txn->snapshot_now->copied);
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e102840..6d65986 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -171,6 +171,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -190,6 +191,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -225,6 +244,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -255,6 +284,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v13-0006-Add-support-for-streaming-to-built-in-replicatio.patch (application/octet-stream, 90.9K) download
From c74472651fd234c3a5fa57d67e0e5fc657f39212 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Apr 2020 11:35:35 +0530
Subject: [PATCH v13 06/10] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 45 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/launcher.c | 1 -
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 140 ++-
src/backend/replication/logical/worker.c | 1026 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 315 +++++-
src/backend/replication/slotfuncs.c | 6 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2033 insertions(+), 43 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 1b8bead..95b7c24 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -164,8 +164,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..3349cc4 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731..f28482f 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 7f15667..65b6b76 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +91,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -175,6 +178,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -318,6 +331,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -334,7 +349,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -412,6 +427,13 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +691,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -697,6 +722,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +740,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +779,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +816,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 50eea2e..4ef4fd4 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4133,6 +4133,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9..5257ab0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -408,6 +408,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..8156a42 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,7 +14,6 @@
*
*-------------------------------------------------------------------------
*/
-
#include "postgres.h"
#include "access/heapam.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 497d8a9..dfc681d 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1148,7 +1148,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1193,7 +1193,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = txn->final_lsn;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd..5242ac0 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,10 +139,15 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -178,8 +183,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -187,6 +192,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -248,7 +257,8 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -256,6 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -296,6 +310,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -305,6 +320,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -347,12 +366,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -397,7 +420,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -405,6 +428,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -685,3 +712,104 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendbyte(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgbyte(in) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a752a12..3dc5f83 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in tmp file directory by default, and the filenames
+ * include both the XID of the toplevel transaction and OID of the subscription.
+ * This is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -33,6 +54,7 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -64,6 +86,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -71,6 +94,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -110,12 +134,57 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
static void apply_handle_insert_internal(ResultRelInfo *relinfo,
EState *estate, TupleTableSlot *remoteslot);
static void apply_handle_update_internal(ResultRelInfo *relinfo,
@@ -187,6 +256,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -553,6 +658,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* XXX optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid, false);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -565,6 +982,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -580,6 +1000,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -616,6 +1039,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -731,6 +1157,9 @@ apply_handle_update(StringInfo s)
RangeTblEntry *target_rte;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -873,6 +1302,9 @@ apply_handle_delete(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -1243,6 +1675,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1381,6 +1816,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1478,6 +1929,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i], true);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1493,6 +1960,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1941,6 +2411,561 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * missing_ok - don't report error for missing file is the flag is passed true.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid, true);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialized in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -2106,6 +3131,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fbf2d4..b0fba26 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -46,29 +46,54 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferChange *change);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-static void send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx);
+static void send_relation_and_attrs(Relation relation, TransactionId xid,
+ LogicalDecodingContext *ctx);
/*
* Entry in the map used to remember which relation schemas we sent.
*
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ *
* For partitions, 'pubactions' considers not only the table's own
* publications, but also those of all of its ancestors.
*/
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
-
+ TransactionId xid; /* transaction that created the record */
/*
* Did we send the schema? If ancestor relid is set, its schema must also
* have been sent for this to be true.
*/
bool schema_sent;
+ List *streamed_txns; /* streamed toplevel transactions with this
+ * schema */
bool replicate_valid;
PublicationActions pubactions;
@@ -94,11 +119,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -114,15 +145,24 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -168,6 +208,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -180,6 +237,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -202,7 +260,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -222,6 +281,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -290,9 +370,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (schema_sent)
return;
/* If needed, send the ancestor's schema first. */
@@ -307,19 +426,25 @@ maybe_send_schema(LogicalDecodingContext *ctx,
oldctx = MemoryContextSwitchTo(CacheMemoryContext);
relentry->map = convert_tuples_by_name(indesc, outdesc);
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, ctx);
- relentry->schema_sent = true;
+ send_relation_and_attrs(relation, xid, ctx);
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
/*
* Sends a relation
*/
static void
-send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx)
+send_relation_and_attrs(Relation relation, TransactionId xid,
+ LogicalDecodingContext *ctx)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -343,17 +468,19 @@ send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx)
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -362,6 +489,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -390,7 +521,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
@@ -410,7 +541,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, xid, relation, tuple);
OutputPluginWrite(ctx, true);
break;
}
@@ -434,7 +565,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple, newtuple);
OutputPluginWrite(ctx, true);
break;
}
@@ -454,7 +585,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, xid, relation, oldtuple);
OutputPluginWrite(ctx, true);
}
else
@@ -479,6 +610,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -507,13 +642,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -587,6 +723,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -623,6 +844,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*
* This looks up publications that the given relation is directly or
@@ -755,6 +1004,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -789,7 +1068,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index f776de3..9121420 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -156,6 +156,12 @@ create_logical_replication_slot(char *name, char *plugin,
NULL);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* If caller needs us to determine the decoding start point, do so now.
* This might take a while.
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 122d884..759ca5c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1004,6 +1004,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..617b909 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b8041d9..3b3e1fd 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -981,7 +981,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561..89158ed 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -86,25 +90,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index f1aa6e9..70d39f8 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v13-0007-Track-statistics-for-streaming.patch (application/octet-stream, 11.8K) download
From 86a29e7801aaae74b8d2b6b5cd9134e7c89ca06e Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 2 Apr 2020 13:19:29 +0530
Subject: [PATCH v13 07/10] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index c50b721..8063ae8 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2063,6 +2063,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d406ea8..65d650d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -788,7 +788,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 7383f14..cfa36b4 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3285,6 +3289,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 759ca5c..1656b4d 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1333,7 +1333,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1354,7 +1354,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2399,6 +2400,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3240,7 +3244,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3297,6 +3301,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3320,6 +3327,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3406,6 +3416,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3654,11 +3669,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4bce3ad..9fb1ffe 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5237,9 +5237,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6d65986..603f325 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -517,15 +517,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f..3888b0c 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index ac31840..68e2deb 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2005,9 +2005,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_slru| SELECT s.name,
s.blks_zeroed,
--
1.8.3.1
v13-0009-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From fbae7eda7cdf0ac9904be1541f196588555241ad Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v13 09/10] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v13-0008-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From 9a8d3be5ee695b2e5fd572304ff3b4ceb40909ae Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v13 08/10] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 6da7f71..086d0c7 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -70,7 +70,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v13-0010-Bugfix-handling-of-incomplete-toast-tuple.patch (application/octet-stream, 14.8K) download
From 909afae31c404497dfab8adc5095f985319d3425 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 28 Feb 2020 11:07:46 +0530
Subject: [PATCH v13 10/10] Bugfix handling of incomplete toast tuple
---
src/backend/access/heap/heapam.c | 3 +
src/backend/replication/logical/decode.c | 17 ++-
src/backend/replication/logical/reorderbuffer.c | 182 +++++++++++++++---------
src/include/access/heapam_xlog.h | 1 +
src/include/replication/reorderbuffer.h | 20 ++-
5 files changed, 147 insertions(+), 76 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index c1586ef..ffa2f0f 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2017,6 +2017,9 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
{
xlrec.flags |= XLH_INSERT_CONTAINS_NEW_TUPLE;
bufflags |= REGBUF_KEEP_DATA;
+
+ if (IsToastRelation(relation))
+ xlrec.flags |= XLH_INSERT_ON_TOAST_RELATION;
}
XLogBeginInsert();
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 4f958e9..e82387c 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -729,7 +729,9 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change,
+ xlrec->flags & XLH_INSERT_ON_TOAST_RELATION);
}
/*
@@ -796,7 +798,8 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -853,7 +856,8 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -889,7 +893,7 @@ DecodeTruncate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
memcpy(change->data.truncate.relids, xlrec->relids,
xlrec->nrelids * sizeof(Oid));
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
}
/*
@@ -989,7 +993,7 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = false;
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
/* move to the next xl_multi_insert_tuple entry */
data += datalen;
@@ -1027,7 +1031,8 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index cfa36b4..fc1c0fb 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -654,11 +654,14 @@ ReorderBufferTXNByXid(ReorderBuffer *rb, TransactionId xid, bool create,
*/
void
ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
- ReorderBufferChange *change)
+ ReorderBufferChange *change, bool toast_insert)
{
- ReorderBufferTXN *txn;
+ ReorderBufferTXN *txn, *toptxn;
+ bool can_stream = false;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ toptxn = txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ if (txn->toptxn != NULL)
+ toptxn = txn->toptxn;
change->lsn = lsn;
change->txn = txn;
@@ -668,9 +671,50 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If this is a toast insert then set the corresponding bit. Otherwise, if
+ * we have toast insert bit set and this is insert/update then clear the
+ * bit.
+ */
+ if (toast_insert)
+ toptxn->txn_flags |= RBTXN_HAS_TOAST_INSERT;
+ else if (rbtxn_has_toast_insert(txn) &&
+ ((change->action == REORDER_BUFFER_CHANGE_INSERT) ||
+ (change->action == REORDER_BUFFER_CHANGE_UPDATE)))
+ {
+ toptxn->txn_flags &= ~RBTXN_HAS_TOAST_INSERT;
+ can_stream = true;
+ }
+
+ /*
+ * If this is a speculative insert then set the corresponding bit.
+ * Otherwise, if we have speculative insert bit set and this is spec confirm
+ * record then clear the bit.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT)
+ toptxn->txn_flags |= RBTXN_HAS_SPEC_INSERT;
+ else if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM)
+ {
+ toptxn->txn_flags &= ~RBTXN_HAS_SPEC_INSERT;
+ can_stream = true;
+ }
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, true);
+ /*
+ * If streaming is enable and we have serialized this transaction because
+ * it had incomplete tuple. So if now we have got the complete tuple we
+ * can stream it.
+ */
+ if (ReorderBufferCanStream(rb) && can_stream && rbtxn_is_serialized(toptxn)
+ && !rbtxn_has_toast_insert(txn) && !rbtxn_has_spec_insert(txn))
+ {
+ ReorderBufferStreamTXN(rb, toptxn);
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
+ }
+
/* check the memory limits and evict something if needed */
ReorderBufferCheckMemoryLimit(rb);
}
@@ -700,7 +744,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
change->data.msg.message = palloc(message_size);
memcpy(change->data.msg.message, message, message_size);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -1862,8 +1906,8 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* disk.
*/
dlist_delete(&change->node);
- ReorderBufferToastAppendChunk(rb, txn, relation,
- change);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
}
change_done:
@@ -2453,7 +2497,7 @@ ReorderBufferAddSnapshot(ReorderBuffer *rb, TransactionId xid,
change->data.snapshot = snap;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2502,7 +2546,7 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
change->data.command_id = cid;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2525,6 +2569,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
Size sz;
ReorderBufferTXN *txn;
+ ReorderBufferTXN *toptxn = NULL;
Assert(change->txn);
@@ -2539,8 +2584,13 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
txn = change->txn;
/* if subxact, and streaming supported, use the toplevel instead */
- if (txn->toptxn && ReorderBufferCanStream(rb))
- txn = txn->toptxn;
+ if (ReorderBufferCanStream(rb))
+ {
+ if (txn->toptxn)
+ toptxn = txn->toptxn;
+ else
+ toptxn = txn;
+ }
sz = ReorderBufferChangeSize(change);
@@ -2548,12 +2598,20 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
txn->size += sz;
rb->size += sz;
+
+ /* Update the total size in the top transaction. */
+ if (toptxn)
+ toptxn->total_size += sz;
}
else
{
Assert((rb->size >= sz) && (txn->size >= sz));
txn->size -= sz;
rb->size -= sz;
+
+ /* Update the total size in the top transaction. */
+ if (toptxn)
+ toptxn->total_size -= sz;
}
Assert(txn->size <= rb->size);
@@ -2620,7 +2678,7 @@ ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
memcpy(change->data.inval.invalidations, msgs,
sizeof(SharedInvalidationMessage) * nmsgs);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -2807,15 +2865,16 @@ ReorderBufferLargestTopTXN(ReorderBuffer *rb)
txn = dlist_container(ReorderBufferTXN, node, iter.cur);
- /* if the current transaction is larger, remember it */
- if ((!largest) || (txn->size > largest->size))
- largest = txn;
+ /*
+ * if the current transaction is larger and doesn't have incomplete data
+ * remember it.
+ */
+ if (((!largest) || (txn->total_size > largest->total_size)) &&
+ ((txn->size > 0) && !rbtxn_has_toast_insert(txn) &&
+ !rbtxn_has_spec_insert(txn)))
+ largest = txn;
}
- Assert(largest);
- Assert(largest->size > 0);
- Assert(largest->size <= rb->size);
-
return largest;
}
@@ -2833,66 +2892,51 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
ReorderBufferTXN *txn;
- /* bail out if we haven't exceeded the memory limit */
- if (rb->size < logical_decoding_work_mem * 1024L)
- return;
-
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by streaming, if supported. Otherwise spill to disk.
- */
- if (ReorderBufferCanStream(rb))
+ /* Loop until we reach under the memory limit. */
+ while (rb->size >= logical_decoding_work_mem * 1024L)
{
/*
- * Pick the largest toplevel transaction and evict it from memory by
- * streaming the already decoded part.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTopTXN(rb);
+ if (ReorderBufferCanStream(rb) &&
+ (txn = ReorderBufferLargestTopTXN(rb)) != NULL)
+ {
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->total_size > 0);
+ Assert(rb->size >= txn->total_size);
- /* we know there has to be one, because the size is not zero */
- Assert(txn && !txn->toptxn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- ReorderBufferStreamTXN(rb, txn);
- }
- else
- {
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
- */
- txn = ReorderBufferLargestTXN(rb);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- /* we know there has to be one, because the size is not zero */
- Assert(txn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
- ReorderBufferSerializeTXN(rb, txn);
+ /*
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
+ */
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
}
- /*
- * After eviction, the transaction should have no entries in memory, and
- * should use 0 bytes for changes.
- *
- * XXX Checking the size is fine for both cases - spill to disk and
- * streaming. But for streaming we should really check nentries_mem for
- * all subtransactions too.
- */
- Assert(txn->size == 0);
- Assert(txn->nentries_mem == 0);
-
- /*
- * And furthermore, evicting the transaction should get us below the
- * memory limit again - it is not possible that we're still exceeding the
- * memory limit after evicting the transaction.
- *
- * This follows from the simple fact that the selected transaction is at
- * least as large as the most recent change (which caused us to go over
- * the memory limit). So by evicting it we're definitely back below the
- * memory limit.
- */
Assert(rb->size < logical_decoding_work_mem * 1024L);
}
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 95d18cd..aa17f7d 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -67,6 +67,7 @@
#define XLH_INSERT_LAST_IN_MULTI (1<<1)
#define XLH_INSERT_IS_SPECULATIVE (1<<2)
#define XLH_INSERT_CONTAINS_NEW_TUPLE (1<<3)
+#define XLH_INSERT_ON_TOAST_RELATION (1<<4)
/*
* xl_heap_update flag values, 8 bits are available.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 603f325..bfc2141 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -172,6 +172,8 @@ typedef struct ReorderBufferChange
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
#define RBTXN_IS_STREAMED 0x0008
+#define RBTXN_HAS_TOAST_INSERT 0x0010
+#define RBTXN_HAS_SPEC_INSERT 0x0020
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -191,6 +193,17 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/* This transaction's changes has toast insert, without main table insert. */
+#define rbtxn_has_toast_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_TOAST_INSERT) != 0 \
+
+/*
+ * This transaction's changes has speculative insert, without speculative
+ * confirm.
+ */
+#define rbtxn_has_spec_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_SPEC_INSERT) != 0 \
+
/*
* Has this transaction been streamed to downstream?
*
@@ -355,6 +368,9 @@ typedef struct ReorderBufferTXN
* Size of this transaction (changes currently in memory, in bytes).
*/
Size size;
+
+ /* Size of top-transaction including sub-transactions. */
+ Size total_size;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -545,7 +561,9 @@ void ReorderBufferReturnChange(ReorderBuffer *, ReorderBufferChange *);
Oid *ReorderBufferGetRelids(ReorderBuffer *, int nrelids);
void ReorderBufferReturnRelids(ReorderBuffer *, Oid *relids);
-void ReorderBufferQueueChange(ReorderBuffer *, TransactionId, XLogRecPtr lsn, ReorderBufferChange *);
+void ReorderBufferQueueChange(ReorderBuffer *, TransactionId,
+ XLogRecPtr lsn, ReorderBufferChange *,
+ bool incomplte_data);
void ReorderBufferQueueMessage(ReorderBuffer *, TransactionId, Snapshot snapshot, XLogRecPtr lsn,
bool transactional, const char *prefix,
Size message_size, const char *message);
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-13 10:44 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Kuntal Ghosh @ 2020-04-13 10:44 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Apr 9, 2020 at 2:40 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have rebased the patch on the latest head. I haven't yet changed
> anything for xid assignment thing because it is not yet concluded.
>
Some review comments from 0001-Immediately-WAL-log-*.patch,
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
IMHO, it's important to reduce the complexity of this function since
it's been called for every WAL insertion. During the lifespan of a
transaction, any of these if conditions will only be evaluated if
previous conditions are true. So, we can maintain some state machine
to avoid multiple evaluation of a condition inside a transaction. But,
if the overhead is not much, it's not worth I guess.
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
This looks wrong. We should change the name of this Macro or we can
add the 1 byte directly in HEADER_SCRATCH_SIZE and some comments.
@@ -195,6 +197,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
The comment looks contradictory.
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
I didn't understand why we need this change in this patch.
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel_xid is valid, we need to assign the subxact to the
+ * toplevel transaction. We need to do this for all records, hence we
+ * do it before the switch.
+ */
s/toplevel_xid/toplevel xid or s/toplevel_xid/txid
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(r->toplevel_xid))
Perhaps, XLogRecGetTopXid() can be used.
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
@ 2020-04-13 11:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-16 21:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-04-13 11:50 UTC (permalink / raw)
To: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Apr 13, 2020 at 4:14 PM Kuntal Ghosh <kuntalghosh.2007@gmail.com> wrote:
>
> On Thu, Apr 9, 2020 at 2:40 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have rebased the patch on the latest head. I haven't yet changed
> > anything for xid assignment thing because it is not yet concluded.
> >
> Some review comments from 0001-Immediately-WAL-log-*.patch,
>
> +bool
> +IsSubTransactionAssignmentPending(void)
> +{
> + if (!XLogLogicalInfoActive())
> + return false;
> +
> + /* we need to be in a transaction state */
> + if (!IsTransactionState())
> + return false;
> +
> + /* it has to be a subtransaction */
> + if (!IsSubTransaction())
> + return false;
> +
> + /* the subtransaction has to have a XID assigned */
> + if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
> + return false;
> +
> + /* and it needs to have 'assigned' */
> + return !CurrentTransactionState->assigned;
> +
> +}
> IMHO, it's important to reduce the complexity of this function since
> it's been called for every WAL insertion. During the lifespan of a
> transaction, any of these if conditions will only be evaluated if
> previous conditions are true. So, we can maintain some state machine
> to avoid multiple evaluation of a condition inside a transaction. But,
> if the overhead is not much, it's not worth I guess.
Yeah maybe, in some cases we can avoid checking multiple conditions by
maintaining that state. But, that state will have to be at the
transaction level. But, I am not sure how much worth it will be to
add one extra condition to skip a few if checks and it will also add
the code complexity. And, in some cases where logical decoding is not
enabled, it may add one extra check? I mean first check the state and
that will take you to the first if check.
>
> +#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
> This looks wrong. We should change the name of this Macro or we can
> add the 1 byte directly in HEADER_SCRATCH_SIZE and some comments.
I think this is in sync with below code (SizeOfXlogOrigin), SO doen't
make much sense to add different terminology no?
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
>
> @@ -195,6 +197,10 @@ XLogResetInsertion(void)
> {
> int i;
>
> + /* reset the subxact assignment flag (if needed) */
> + if (curinsert_flags & XLOG_INCLUDE_XID)
> + MarkSubTransactionAssigned();
> The comment looks contradictory.
>
> XLogSetRecordFlags(uint8 flags)
> {
> Assert(begininsert_called);
> - curinsert_flags = flags;
> + curinsert_flags |= flags;
> }
> I didn't understand why we need this change in this patch.
I think it's changed so that below code can use it, but we have
directly set the flag. I think I will change in the next version.
@@ -748,6 +754,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ curinsert_flags |= XLOG_INCLUDE_XID;
>
> + txid = XLogRecGetTopXid(record);
> +
> + /*
> + * If the toplevel_xid is valid, we need to assign the subxact to the
> + * toplevel transaction. We need to do this for all records, hence we
> + * do it before the switch.
> + */
> s/toplevel_xid/toplevel xid or s/toplevel_xid/txid
Okay, we can change
> if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
> - info != XLOG_XACT_ASSIGNMENT)
> + !TransactionIdIsValid(r->toplevel_xid))
> Perhaps, XLogRecGetTopXid() can be used.
ok
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-13 12:41 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 09:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Kuntal Ghosh @ 2020-04-13 12:41 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Apr 13, 2020 at 5:20 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> On Mon, Apr 13, 2020 at 4:14 PM Kuntal Ghosh <kuntalghosh.2007@gmail.com> wrote:
> >
> > +#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
> > This looks wrong. We should change the name of this Macro or we can
> > add the 1 byte directly in HEADER_SCRATCH_SIZE and some comments.
>
> I think this is in sync with below code (SizeOfXlogOrigin), SO doen't
> make much sense to add different terminology no?
> #define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
> +#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
>
In that case, we can rename this, for example, SizeOfXLogTransactionId.
Some review comments from 0002-Issue-individual-*.path,
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, int nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ /* XXX Should we even write invalidations without valid XID? */
+ if (xid == InvalidTransactionId)
+ return;
+
+ Assert(xid != InvalidTransactionId);
It seems we don't call the function if xid is not valid. In fact,
@@ -281,6 +281,24 @@ DecodeXactOp(LogicalDecodingContext *ctx,
XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ if (!TransactionIdIsValid(xid))
+ break;
+
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->nmsgs, invals->msgs);
Why should we insert an WAL record for such cases?
+ * When wal_level=logical, write invalidations into WAL at each command end to
+ * support the decoding of the in-progress transaction. As of now it was
+ * enough to log invalidation only at commit because we are only decoding the
+ * transaction at the commit time. We only need to log the catalog cache and
+ * relcache invalidation. There can not be any active MVCC scan in logical
+ * decoding so we don't need to log the snapshot invalidation.
The alignment is not right.
/*
* CommandEndInvalidationMessages
- * Process queued-up invalidation messages at end of one command
- * in a transaction.
+ * Process queued-up invalidation messages at end of one command
+ * in a transaction.
Looks unnecessary changes.
* Note:
- * This should be called during CommandCounterIncrement(),
- * after we have advanced the command ID.
+ * This should be called during CommandCounterIncrement(),
+ * after we have advanced the command ID.
*/
Looks unnecessary changes.
if (transInvalInfo == NULL)
- return;
+ return;
Looks unnecessary changes.
+ /* prepare record */
+ memset(&xlrec, 0, sizeof(xlrec));
We should use MinSizeOfXactInvalidations, no?
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
@ 2020-04-13 13:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-04-13 13:04 UTC (permalink / raw)
To: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Apr 13, 2020 at 6:12 PM Kuntal Ghosh <kuntalghosh.2007@gmail.com> wrote:
>
> On Mon, Apr 13, 2020 at 5:20 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > On Mon, Apr 13, 2020 at 4:14 PM Kuntal Ghosh <kuntalghosh.2007@gmail.com> wrote:
> > >
> > > +#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
> > > This looks wrong. We should change the name of this Macro or we can
> > > add the 1 byte directly in HEADER_SCRATCH_SIZE and some comments.
> >
> > I think this is in sync with below code (SizeOfXlogOrigin), SO doen't
> > make much sense to add different terminology no?
> > #define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
> > +#define SizeOfTransactionId (sizeof(TransactionId) + sizeof(char))
> >
> In that case, we can rename this, for example, SizeOfXLogTransactionId.
Make sense.
>
> Some review comments from 0002-Issue-individual-*.path,
>
> +void
> +ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
> + XLogRecPtr lsn, int nmsgs,
> + SharedInvalidationMessage *msgs)
> +{
> + MemoryContext oldcontext;
> + ReorderBufferChange *change;
> +
> + /* XXX Should we even write invalidations without valid XID? */
> + if (xid == InvalidTransactionId)
> + return;
> +
> + Assert(xid != InvalidTransactionId);
>
> It seems we don't call the function if xid is not valid. In fact,
>
> @@ -281,6 +281,24 @@ DecodeXactOp(LogicalDecodingContext *ctx,
> XLogRecordBuffer *buf)
> }
> case XLOG_XACT_ASSIGNMENT:
> break;
> + case XLOG_XACT_INVALIDATIONS:
> + {
> + TransactionId xid;
> + xl_xact_invalidations *invals;
> +
> + xid = XLogRecGetXid(r);
> + invals = (xl_xact_invalidations *) XLogRecGetData(r);
> +
> + if (!TransactionIdIsValid(xid))
> + break;
> +
> + ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
> + invals->nmsgs, invals->msgs);
>
> Why should we insert an WAL record for such cases?
I think we can avoid this. I will analyze and send update in my next patch.
>
> + * When wal_level=logical, write invalidations into WAL at each command end to
> + * support the decoding of the in-progress transaction. As of now it was
> + * enough to log invalidation only at commit because we are only decoding the
> + * transaction at the commit time. We only need to log the catalog cache and
> + * relcache invalidation. There can not be any active MVCC scan in logical
> + * decoding so we don't need to log the snapshot invalidation.
> The alignment is not right.
Will fix.
> /*
> * CommandEndInvalidationMessages
> - * Process queued-up invalidation messages at end of one command
> - * in a transaction.
> + * Process queued-up invalidation messages at end of one command
> + * in a transaction.
> Looks unnecessary changes.
Will fix.
>
> * Note:
> - * This should be called during CommandCounterIncrement(),
> - * after we have advanced the command ID.
> + * This should be called during CommandCounterIncrement(),
> + * after we have advanced the command ID.
> */
> Looks unnecessary changes.
Will fix.
> if (transInvalInfo == NULL)
> - return;
> + return;
> Looks unnecessary changes.
>
> + /* prepare record */
> + memset(&xlrec, 0, sizeof(xlrec));
> We should use MinSizeOfXactInvalidations, no?
Right.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-13 18:13 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Kuntal Ghosh @ 2020-04-13 18:13 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Apr 13, 2020 at 6:34 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
Skipping 0003 for now. Review comments from 0004-Gracefully-handle-*.patch
@@ -5490,6 +5523,14 @@ heap_finish_speculative(Relation relation,
ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_hot_search with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search call during logical decoding");
The call is to heap_finish_speculative.
@@ -481,6 +482,19 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
s/transaction aborted/transaction aborted concurrently perhaps? Also,
can we move this check at the begining of the function? If the
condition fails, we can skip the sys scan.
Some of the checks looks repetative in the same file. Should we
declare them as inline functions?
Review comments from 0005-Implement-streaming*.patch
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
...
+#endif
+}
We can implement the same as following:
#ifdef USE_ASSERT_CHECKING
static void
AssertChangeLsnOrder(ReorderBufferTXN *txn)
{
dlist_iter iter;
...
}
#else
#define AssertChangeLsnOrder(txn) ((void)true)
#endif
+ * if it is aborted we will report an specific error which we can ignore. We
s/an specific/a specific
+ * Set the last last of the stream as the final lsn before calling
+ * stream stop.
s/last last/last
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
When we don't re-throw, the errdata should be freed by calling
FreeErrorData(errdata), right?
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+
+ FlushErrorState();
+ }
stream_stop() can still throw some error, right? In that case, we
should flush the error state before calling stream_stop().
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
Hmm, it seems this part needs an assumption that after copying the
snapshot, no subsequent step can throw any error. If they do, then we
can again create a copy of the snapshot in catch block, which will
leak some memory. Is my understanding correct?
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ PG_RE_THROW();
+ }
Shouldn't we switch back to previously created error memory context
before re-throwing?
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id = FirstCommandId;
In the modified ReorderBufferCommit(), why is it necessary to declare
the above two variable as volatile? There is no try-catch block here.
@@ -1946,6 +2284,13 @@ ReorderBufferAbort(ReorderBuffer *rb,
TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
s/When/If
+ /*
+ * When the (sub)transaction was streamed, notify the remote node
+ * about the abort.
+ */
+ if (rbtxn_is_streamed(txn))
+ rb->stream_abort(rb, txn, lsn);
s/When/If. And, in this case, if we've not sent any data, why should
we send the abort message (similar to the previous one)?
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
Should we put any assert (not necessarily here) to validate the above comment?
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
The same three assertions are already there in ReorderBufferLargestTopTXN().
+static bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
Potential inline function.
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ volatile Snapshot snapshot_now;
+ volatile CommandId command_id;
Here also, do we need to declare these two variables as volatile?
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
@ 2020-04-14 10:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 20:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-04-14 10:10 UTC (permalink / raw)
To: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Apr 13, 2020 at 11:43 PM Kuntal Ghosh
<kuntalghosh.2007@gmail.com> wrote:
>
> On Mon, Apr 13, 2020 at 6:34 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> Skipping 0003 for now. Review comments from 0004-Gracefully-handle-*.patch
>
> @@ -5490,6 +5523,14 @@ heap_finish_speculative(Relation relation,
> ItemPointer tid)
> ItemId lp = NULL;
> HeapTupleHeader htup;
>
> + /*
> + * We don't expect direct calls to heap_hot_search with
> + * valid CheckXidAlive for regular tables. Track that below.
> + */
> + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> + !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
> + elog(ERROR, "unexpected heap_hot_search call during logical decoding");
> The call is to heap_finish_speculative.
Fixed
> @@ -481,6 +482,19 @@ systable_getnext(SysScanDesc sysscan)
> }
> }
>
> + if (TransactionIdIsValid(CheckXidAlive) &&
> + !TransactionIdIsInProgress(CheckXidAlive) &&
> + !TransactionIdDidCommit(CheckXidAlive))
> + ereport(ERROR,
> + (errcode(ERRCODE_TRANSACTION_ROLLBACK),
> + errmsg("transaction aborted during system catalog scan")));
> s/transaction aborted/transaction aborted concurrently perhaps? Also,
> can we move this check at the begining of the function? If the
> condition fails, we can skip the sys scan.
We must check this after we get the tuple because our goal is, not to
decode based on the wrong tuple. And, if we move the check before
then, what if the transaction aborted after the check. Once we get
the tuple and if the transaction is alive by that time then it doesn't
matter even if it aborts because we have got the right tuple already.
>
> Some of the checks looks repetative in the same file. Should we
> declare them as inline functions?
>
> Review comments from 0005-Implement-streaming*.patch
>
> +static void
> +AssertChangeLsnOrder(ReorderBufferTXN *txn)
> +{
> +#ifdef USE_ASSERT_CHECKING
> + dlist_iter iter;
> ...
> +#endif
> +}
>
> We can implement the same as following:
> #ifdef USE_ASSERT_CHECKING
> static void
> AssertChangeLsnOrder(ReorderBufferTXN *txn)
> {
> dlist_iter iter;
> ...
> }
> #else
> #define AssertChangeLsnOrder(txn) ((void)true)
> #endif
I am not sure, this doesn't look clean. Moreover, the other similar
functions are defined in the same way. e.g. AssertTXNLsnOrder.
>
> + * if it is aborted we will report an specific error which we can ignore. We
> s/an specific/a specific
Done
>
> + * Set the last last of the stream as the final lsn before calling
> + * stream stop.
> s/last last/last
>
> PG_CATCH();
> {
> + MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
> + ErrorData *errdata = CopyErrorData();
> When we don't re-throw, the errdata should be freed by calling
> FreeErrorData(errdata), right?
Done
>
> + /*
> + * Set the last last of the stream as the final lsn before
> + * calling stream stop.
> + */
> + txn->final_lsn = prev_lsn;
> + rb->stream_stop(rb, txn);
> +
> + FlushErrorState();
> + }
> stream_stop() can still throw some error, right? In that case, we
> should flush the error state before calling stream_stop().
Done
>
> + /*
> + * Remember the command ID and snapshot if transaction is streaming
> + * otherwise free the snapshot if we have copied it.
> + */
> + if (streaming)
> + {
> + txn->command_id = command_id;
> +
> + /* Avoid copying if it's already copied. */
> + if (snapshot_now->copied)
> + txn->snapshot_now = snapshot_now;
> + else
> + txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
> + txn, command_id);
> + }
> + else if (snapshot_now->copied)
> + ReorderBufferFreeSnap(rb, snapshot_now);
> Hmm, it seems this part needs an assumption that after copying the
> snapshot, no subsequent step can throw any error. If they do, then we
> can again create a copy of the snapshot in catch block, which will
> leak some memory. Is my understanding correct?
Actually, In CATCH we copy only if the error is
ERRCODE_TRANSACTION_ROLLBACK. And, that can occur during systable
scan. Basically, in TRY block we copy snapshot after we have streamed
all the changes i.e. systable scan is done, now if there is any error
that will not be ERRCODE_TRANSACTION_ROLLBACK. So we will not copy
again.
>
> + }
> + else
> + {
> + ReorderBufferCleanupTXN(rb, txn);
> + PG_RE_THROW();
> + }
> Shouldn't we switch back to previously created error memory context
> before re-throwing?
Fixed.
>
> +ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
> + XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
> + TimestampTz commit_time,
> + RepOriginId origin_id, XLogRecPtr origin_lsn)
> +{
> + ReorderBufferTXN *txn;
> + volatile Snapshot snapshot_now;
> + volatile CommandId command_id = FirstCommandId;
> In the modified ReorderBufferCommit(), why is it necessary to declare
> the above two variable as volatile? There is no try-catch block here.
Fixed
>
> @@ -1946,6 +2284,13 @@ ReorderBufferAbort(ReorderBuffer *rb,
> TransactionId xid, XLogRecPtr lsn)
> if (txn == NULL)
> return;
>
> + /*
> + * When the (sub)transaction was streamed, notify the remote node
> + * about the abort only if we have sent any data for this transaction.
> + */
> + if (rbtxn_is_streamed(txn) && txn->any_data_sent)
> + rb->stream_abort(rb, txn, lsn);
> +
> s/When/If
>
> + /*
> + * When the (sub)transaction was streamed, notify the remote node
> + * about the abort.
> + */
> + if (rbtxn_is_streamed(txn))
> + rb->stream_abort(rb, txn, lsn);
> s/When/If. And, in this case, if we've not sent any data, why should
> we send the abort message (similar to the previous one)?
Fixed
>
> + * Note: We never do both stream and serialize a transaction (we only spill
> + * to disk when streaming is not supported by the plugin), so only one of
> + * those two flags may be set at any given time.
> + */
> +#define rbtxn_is_streamed(txn) \
> +( \
> + ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
> +)
> Should we put any assert (not necessarily here) to validate the above comment?
Because of toast handling, this assumption is changed now so I will
remove this note in that patch (0010).
>
> + txn = ReorderBufferLargestTopTXN(rb);
> +
> + /* we know there has to be one, because the size is not zero */
> + Assert(txn && !txn->toptxn);
> + Assert(txn->size > 0);
> + Assert(rb->size >= txn->size);
> The same three assertions are already there in ReorderBufferLargestTopTXN().
>
> +static bool
> +ReorderBufferCanStream(ReorderBuffer *rb)
> +{
> + LogicalDecodingContext *ctx = rb->private_data;
> +
> + return ctx->streaming;
> +}
> Potential inline function.
Done
> +static void
> +ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
> +{
> + volatile Snapshot snapshot_now;
> + volatile CommandId command_id;
> Here also, do we need to declare these two variables as volatile?
Done
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v14-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.5K) download
From 4e1d4a1d40ae55d5144f73ed7f607b1ad22a3106 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 20 Mar 2020 15:03:01 +0530
Subject: [PATCH v14 01/10] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 46 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 +++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 ++++++++++++++-------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 99 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 6b1ae1f..c5842d3 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -190,6 +190,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -222,6 +223,7 @@ typedef struct SerializedTransactionState
static TransactionStateData TopTransactionStateData = {
.state = TRANS_DEFAULT,
.blockState = TBLOCK_DEFAULT,
+ .assigned = false,
};
/*
@@ -5111,6 +5113,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6013,3 +6016,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 4259309..3c49954 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -89,11 +89,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfXLogTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfXLogTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -195,6 +197,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -398,7 +404,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -748,6 +754,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ XLogSetRecordFlags(XLOG_INCLUDE_XID);
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 79ff976..7b5257f 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1189,6 +1189,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1227,6 +1228,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index c2e5e3a..122c581 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel xid is valid, we need to assign the subxact to the
+ * toplevel xact. We need to do this for all records, hence we do it before
+ * the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ txid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrId) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(XLogRecGetTopXid(r)))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index f60ed2d..6d439d0 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -229,6 +229,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 4582196..756f6df 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -150,6 +150,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -285,6 +287,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From 9416517a4e1396f2ce5ac040ada88ba59e4f183d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v14 03/10] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 93c9488..64f651f 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -540,3 +569,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index bad3bfe..65244b1 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 5adf253..497d8a9 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -862,6 +910,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index af35287..e102840 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -354,6 +354,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -393,6 +439,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v14-0002-Issue-individual-invalidations-with.patch (application/octet-stream, 16.7K) download
From b0ed16bb5a75a08860b18f0212c4445b60dcb382 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Tue, 14 Apr 2020 11:11:37 +0530
Subject: [PATCH v14 02/10] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
---
src/backend/access/rmgrdesc/xactdesc.c | 40 +++++++++
src/backend/access/transam/xact.c | 7 ++
src/backend/replication/logical/decode.c | 16 ++++
src/backend/replication/logical/reorderbuffer.c | 104 +++++++++++++++++++++---
src/backend/utils/cache/inval.c | 49 +++++++++++
src/include/access/xact.h | 13 ++-
src/include/replication/reorderbuffer.h | 11 +++
7 files changed, 226 insertions(+), 14 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index fbc5942..17c06f7 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,9 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -396,6 +399,12 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs);
+ }
}
const char *
@@ -423,7 +432,38 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs)
+{
+ int i;
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index c5842d3..cf78ffc 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6013,6 +6013,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 122c581..69c1f45 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,22 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ Assert(TransactionIdIsValid(xid));
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->nmsgs, invals->msgs);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 4594cf9..0d5bb73 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -220,7 +220,7 @@ static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
-static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs);
/*
* ---------------------------------------
@@ -455,6 +455,11 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
pfree(change->data.msg.message);
change->data.msg.message = NULL;
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ if (change->data.inval.invalidations)
+ pfree(change->data.inval.invalidations);
+ change->data.inval.invalidations = NULL;
+ break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
if (change->data.snapshot)
{
@@ -1814,17 +1819,24 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation messages locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ ReorderBufferExecuteInvalidations(
+ change->data.inval.ninvalidations,
+ change->data.inval.invalidations);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -1866,7 +1878,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -1892,7 +1905,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2204,6 +2218,33 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, int nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ MemoryContextAlloc(rb->context,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2234,12 +2275,12 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
* in the changestream but we don't know which those are.
*/
static void
-ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs)
{
int i;
- for (i = 0; i < txn->ninvalidations; i++)
- LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+ for (i = 0; i < nmsgs; i++)
+ LocalExecuteInvalidationMessage(&msgs[i]);
}
/*
@@ -2591,6 +2632,24 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ {
+ char *data;
+ Size inval_size = sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+
+ sz += inval_size;
+
+ ReorderBufferSerializeReserve(rb, sz);
+ data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
+
+ /* might have been reallocated above */
+ ondisk = (ReorderBufferDiskChange *) rb->outbuf;
+ memcpy(data, change->data.inval.invalidations, inval_size);
+ data += inval_size;
+
+ break;
+ }
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot snap;
@@ -2736,6 +2795,11 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ sz += sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot snap;
@@ -3004,6 +3068,20 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ {
+ Size inval_size = sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+
+ change->data.inval.invalidations =
+ MemoryContextAlloc(rb->context,
+ change->data.msg.message_size);
+ /* read the message */
+ memcpy(change->data.inval.invalidations, data, inval_size);
+ data += inval_size;
+
+ break;
+ }
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot oldsnap;
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..cba5b6c 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write invalidations into WAL at each command end to
+ * support the decoding of the in-progress transaction. As of now it was
+ * enough to log invalidation only at commit because we are only decoding the
+ * transaction at the commit time. We only need to log the catalog cache and
+ * relcache invalidation. There can not be any active MVCC scan in logical
+ * decoding so we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,8 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(void);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -1092,6 +1101,9 @@ CommandEndInvalidationMessages(void)
if (transInvalInfo == NULL)
return;
+ if (XLogLogicalInfoActive())
+ LogLogicalInvalidations();
+
ProcessInvalidationMessages(&transInvalInfo->CurrentCmdInvalidMsgs,
LocalExecuteInvalidationMessage);
AppendInvalidationMessages(&transInvalInfo->PriorCmdInvalidMsgs,
@@ -1501,3 +1513,40 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations()
+{
+ xl_xact_invalidations xlrec;
+ SharedInvalidationMessage *invalMessages;
+ int nmsgs = 0;
+
+ if (transInvalInfo->CurrentCmdInvalidMsgs.cclist)
+ {
+ ProcessInvalidationMessagesMulti(&transInvalInfo->CurrentCmdInvalidMsgs,
+ MakeSharedInvalidMessagesArray);
+ invalMessages = SharedInvalidMessagesArray;
+ nmsgs = numSharedInvalidMessagesArray;
+ SharedInvalidMessagesArray = NULL;
+ numSharedInvalidMessagesArray = 0;
+ }
+
+ if (nmsgs > 0)
+ {
+ /* prepare record */
+ memset(&xlrec, 0, MinSizeOfXactInvalidations);
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) invalMessages,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+
+ pfree(invalMessages);
+ }
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..b822c5e 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,17 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ */
+typedef struct xl_xact_invalidations
+{
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 626ecf4..af35287 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,14 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ uint32 ninvalidations; /* Number of messages */
+ SharedInvalidationMessage *invalidations; /* invalidation
+ * message */
+ } inval;
} data;
/*
@@ -459,6 +468,8 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ int nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 37.9K) download
From 75427813941c3e8d1af01f7f2800da9994746c5b Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v14 05/10] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 713 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 36 ++
3 files changed, 696 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 2302875..801fdc5 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static inline bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -773,6 +786,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -865,6 +910,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -988,7 +1036,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1024,6 +1072,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1038,6 +1089,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1316,6 +1370,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1341,8 +1404,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1350,9 +1498,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1491,63 +1636,75 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+/*
+ * Set xid for concurrent abort check
+ *
+ * While streaming an in-progress transaction there is a possibility that the
+ * (sub)transaction might get aborted concurrently. In such case if the
+ * (sub)transaction has catalog update then we might decode the tuple using
+ * wrong catalog version. So for detecting the concurrent abort we set
+ * CheckXidAlive to the current (sub)transaction's xid for which this change
+ * belongs to. And, during catalog scan we can check the status of the xid and
+ * if it is aborted we will report a specific error that we can ignore. We
+ * might have already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the abort we will
+ * stream abort message to truncate the changes in the subscriber.
+ */
+static inline void
+SetupCheckXidLive(TransactionId xid)
+{
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
+ if (!TransactionIdDidCommit(xid))
+ CheckXidAlive = xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
- snapshot_now = txn->base_snapshot;
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
- /* build data to be able to lookup the CommandIds of catalog tuples */
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1563,15 +1720,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1579,6 +1741,19 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* Set the xid for concurrent abort check. */
+ if (streaming)
+ SetupCheckXidLive(change->txn->xid);
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1588,8 +1763,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1655,7 +1828,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1676,8 +1857,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1695,7 +1874,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1753,7 +1932,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1762,10 +1949,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1796,9 +1989,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1818,7 +2011,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1858,14 +2052,46 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ if (!XLogRecPtrIsInvalid(prev_lsn))
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1884,14 +2110,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,18 +2145,125 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ Snapshot snapshot_now;
+ CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1946,6 +2287,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * If the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2015,6 +2363,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * If the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2150,8 +2505,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2159,6 +2523,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2170,19 +2535,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2211,6 +2585,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2295,6 +2670,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2399,6 +2781,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2418,15 +2832,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- ReorderBufferSerializeTXN(rb, txn);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2746,6 +3191,102 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static inline bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after the last
+ * streaming run, we might have got some new sub-transactions. So we
+ * need to add them to the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ Assert(txn->snapshot_now->copied);
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e102840..6d65986 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -171,6 +171,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -190,6 +191,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -225,6 +244,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -255,6 +284,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch (application/octet-stream, 12.3K) download
From 0d995d97213750c321f7ccfbe6d184ccd335fb52 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Apr 2020 10:55:19 +0530
Subject: [PATCH v14 04/10] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 38 +++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++++--
src/include/utils/snapmgr.h | 4 ++-
6 files changed, 113 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 65244b1..b59a6c8 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>pg_current_xact_id()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index c4a5aa6..86c2190 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1303,6 +1303,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
HEAPDEBUG_1; /* heap_getnext( info ) */
@@ -1422,6 +1431,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1536,6 +1553,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1685,6 +1710,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5490,6 +5523,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_finish_speculative with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_finish_speculative call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index dfba5ae..97a1075 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -433,6 +434,25 @@ systable_beginscan(Relation heapRelation,
}
/*
+ * HandleConcurrentAbort - Handle concurrent abort of the CheckXidAlive.
+ *
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we error
+ * out. We can't directly use TransactionIdDidAbort as after crash such
+ * transaction might not have been marked as aborted. See detailed comments at
+ * snapmgr.c where the variable is declared.
+ */
+static inline void
+HandleConcurrentAbort()
+{
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+}
+
+/*
* systable_getnext --- get next tuple in a heap-or-index scan
*
* Returns NULL if no more tuples available.
@@ -481,6 +501,12 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * Handle the concurrent abort while fetching the catalog tuple during
+ * logical streaming of a transaction.
+ */
+ HandleConcurrentAbort();
+
return htup;
}
@@ -517,6 +543,12 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * Handle the concurrent abort while fetching the catalog tuple during
+ * logical streaming of a transaction.
+ */
+ HandleConcurrentAbort();
+
return result;
}
@@ -643,6 +675,12 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * Handle the concurrent abort while fetching the catalog tuple during
+ * logical streaming of a transaction.
+ */
+ HandleConcurrentAbort();
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0d5bb73..2302875 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -696,7 +696,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1547,7 +1547,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1798,7 +1798,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1818,7 +1818,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch (application/octet-stream, 90.9K) download
From 884ef0f05d48ec80692769b92cd648d989cdb2e0 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Apr 2020 11:35:35 +0530
Subject: [PATCH v14 06/10] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 45 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/launcher.c | 1 -
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 140 ++-
src/backend/replication/logical/worker.c | 1026 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 315 +++++-
src/backend/replication/slotfuncs.c | 6 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2033 insertions(+), 43 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 1b8bead..95b7c24 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -164,8 +164,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..3349cc4 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731..f28482f 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 7f15667..65b6b76 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +91,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -175,6 +178,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -318,6 +331,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -334,7 +349,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -412,6 +427,13 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +691,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -697,6 +722,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +740,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +779,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +816,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 50eea2e..4ef4fd4 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4133,6 +4133,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9..5257ab0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -408,6 +408,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..8156a42 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,7 +14,6 @@
*
*-------------------------------------------------------------------------
*/
-
#include "postgres.h"
#include "access/heapam.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 497d8a9..dfc681d 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1148,7 +1148,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1193,7 +1193,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = txn->final_lsn;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd..5242ac0 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,10 +139,15 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -178,8 +183,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -187,6 +192,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -248,7 +257,8 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -256,6 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -296,6 +310,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -305,6 +320,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -347,12 +366,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -397,7 +420,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -405,6 +428,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -685,3 +712,104 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendbyte(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgbyte(in) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a752a12..3dc5f83 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in tmp file directory by default, and the filenames
+ * include both the XID of the toplevel transaction and OID of the subscription.
+ * This is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -33,6 +54,7 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -64,6 +86,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -71,6 +94,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -110,12 +134,57 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
static void apply_handle_insert_internal(ResultRelInfo *relinfo,
EState *estate, TupleTableSlot *remoteslot);
static void apply_handle_update_internal(ResultRelInfo *relinfo,
@@ -187,6 +256,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -553,6 +658,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* XXX optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid, false);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -565,6 +982,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -580,6 +1000,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -616,6 +1039,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -731,6 +1157,9 @@ apply_handle_update(StringInfo s)
RangeTblEntry *target_rte;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -873,6 +1302,9 @@ apply_handle_delete(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -1243,6 +1675,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1381,6 +1816,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1478,6 +1929,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i], true);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1493,6 +1960,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1941,6 +2411,561 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * missing_ok - don't report error for missing file is the flag is passed true.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid, true);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialized in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -2106,6 +3131,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 5fbf2d4..b0fba26 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -46,29 +46,54 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferChange *change);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-static void send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx);
+static void send_relation_and_attrs(Relation relation, TransactionId xid,
+ LogicalDecodingContext *ctx);
/*
* Entry in the map used to remember which relation schemas we sent.
*
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ *
* For partitions, 'pubactions' considers not only the table's own
* publications, but also those of all of its ancestors.
*/
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
-
+ TransactionId xid; /* transaction that created the record */
/*
* Did we send the schema? If ancestor relid is set, its schema must also
* have been sent for this to be true.
*/
bool schema_sent;
+ List *streamed_txns; /* streamed toplevel transactions with this
+ * schema */
bool replicate_valid;
PublicationActions pubactions;
@@ -94,11 +119,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -114,15 +145,24 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -168,6 +208,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -180,6 +237,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -202,7 +260,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -222,6 +281,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -290,9 +370,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (schema_sent)
return;
/* If needed, send the ancestor's schema first. */
@@ -307,19 +426,25 @@ maybe_send_schema(LogicalDecodingContext *ctx,
oldctx = MemoryContextSwitchTo(CacheMemoryContext);
relentry->map = convert_tuples_by_name(indesc, outdesc);
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, ctx);
- relentry->schema_sent = true;
+ send_relation_and_attrs(relation, xid, ctx);
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
/*
* Sends a relation
*/
static void
-send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx)
+send_relation_and_attrs(Relation relation, TransactionId xid,
+ LogicalDecodingContext *ctx)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -343,17 +468,19 @@ send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx)
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -362,6 +489,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -390,7 +521,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
@@ -410,7 +541,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, xid, relation, tuple);
OutputPluginWrite(ctx, true);
break;
}
@@ -434,7 +565,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple, newtuple);
OutputPluginWrite(ctx, true);
break;
}
@@ -454,7 +585,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, xid, relation, oldtuple);
OutputPluginWrite(ctx, true);
}
else
@@ -479,6 +610,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -507,13 +642,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -587,6 +723,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -623,6 +844,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*
* This looks up publications that the given relation is directly or
@@ -755,6 +1004,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -789,7 +1068,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index f776de3..9121420 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -156,6 +156,12 @@ create_logical_replication_slot(char *name, char *plugin,
NULL);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* If caller needs us to determine the decoding start point, do so now.
* This might take a while.
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 122d884..759ca5c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1004,6 +1004,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..617b909 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b8041d9..3b3e1fd 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -981,7 +981,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561..89158ed 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -86,25 +90,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index f1aa6e9..70d39f8 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From cb85389c81638ea84e8f6cea006df7648ec6deb6 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v14 08/10] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 6da7f71..086d0c7 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -70,7 +70,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From 31a85235bcb18038a86b2e57a9cdcf2d893de27b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v14 09/10] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v14-0007-Track-statistics-for-streaming.patch (application/octet-stream, 11.8K) download
From be833105e8703d7471db400def123b4c121ac71b Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 2 Apr 2020 13:19:29 +0530
Subject: [PATCH v14 07/10] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index c50b721..8063ae8 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2063,6 +2063,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d406ea8..65d650d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -788,7 +788,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 801fdc5..845d820 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3282,6 +3286,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 759ca5c..1656b4d 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1333,7 +1333,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1354,7 +1354,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2399,6 +2400,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3240,7 +3244,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3297,6 +3301,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
@@ -3320,6 +3327,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
memset(nulls, 0, sizeof(nulls));
@@ -3406,6 +3416,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3654,11 +3669,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4bce3ad..9fb1ffe 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5237,9 +5237,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6d65986..603f325 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -517,15 +517,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 366828f..3888b0c 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -85,6 +85,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index ac31840..68e2deb 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2005,9 +2005,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_slru| SELECT s.name,
s.blks_zeroed,
--
1.8.3.1
v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch (application/octet-stream, 15.3K) download
From 0054a3e3cf50db92317e1fec0583f3777b1f8a32 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 28 Feb 2020 11:07:46 +0530
Subject: [PATCH v14 10/10] Bugfix handling of incomplete toast tuple
---
src/backend/access/heap/heapam.c | 3 +
src/backend/replication/logical/decode.c | 17 ++-
src/backend/replication/logical/reorderbuffer.c | 182 +++++++++++++++---------
src/include/access/heapam_xlog.h | 1 +
src/include/replication/reorderbuffer.h | 24 +++-
5 files changed, 147 insertions(+), 80 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 86c2190..8fca8cb 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2017,6 +2017,9 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
{
xlrec.flags |= XLH_INSERT_CONTAINS_NEW_TUPLE;
bufflags |= REGBUF_KEEP_DATA;
+
+ if (IsToastRelation(relation))
+ xlrec.flags |= XLH_INSERT_ON_TOAST_RELATION;
}
XLogBeginInsert();
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 69c1f45..c841687 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -727,7 +727,9 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change,
+ xlrec->flags & XLH_INSERT_ON_TOAST_RELATION);
}
/*
@@ -794,7 +796,8 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -851,7 +854,8 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -887,7 +891,7 @@ DecodeTruncate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
memcpy(change->data.truncate.relids, xlrec->relids,
xlrec->nrelids * sizeof(Oid));
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
}
/*
@@ -987,7 +991,7 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = false;
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
/* move to the next xl_multi_insert_tuple entry */
data += datalen;
@@ -1025,7 +1029,8 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 845d820..f299c64 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -654,11 +654,14 @@ ReorderBufferTXNByXid(ReorderBuffer *rb, TransactionId xid, bool create,
*/
void
ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
- ReorderBufferChange *change)
+ ReorderBufferChange *change, bool toast_insert)
{
- ReorderBufferTXN *txn;
+ ReorderBufferTXN *txn, *toptxn;
+ bool can_stream = false;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ toptxn = txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ if (txn->toptxn != NULL)
+ toptxn = txn->toptxn;
change->lsn = lsn;
change->txn = txn;
@@ -668,9 +671,50 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If this is a toast insert then set the corresponding bit. Otherwise, if
+ * we have toast insert bit set and this is insert/update then clear the
+ * bit.
+ */
+ if (toast_insert)
+ toptxn->txn_flags |= RBTXN_HAS_TOAST_INSERT;
+ else if (rbtxn_has_toast_insert(txn) &&
+ ((change->action == REORDER_BUFFER_CHANGE_INSERT) ||
+ (change->action == REORDER_BUFFER_CHANGE_UPDATE)))
+ {
+ toptxn->txn_flags &= ~RBTXN_HAS_TOAST_INSERT;
+ can_stream = true;
+ }
+
+ /*
+ * If this is a speculative insert then set the corresponding bit.
+ * Otherwise, if we have speculative insert bit set and this is spec confirm
+ * record then clear the bit.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT)
+ toptxn->txn_flags |= RBTXN_HAS_SPEC_INSERT;
+ else if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM)
+ {
+ toptxn->txn_flags &= ~RBTXN_HAS_SPEC_INSERT;
+ can_stream = true;
+ }
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, true);
+ /*
+ * If streaming is enable and we have serialized this transaction because
+ * it had incomplete tuple. So if now we have got the complete tuple we
+ * can stream it.
+ */
+ if (ReorderBufferCanStream(rb) && can_stream && rbtxn_is_serialized(toptxn)
+ && !rbtxn_has_toast_insert(txn) && !rbtxn_has_spec_insert(txn))
+ {
+ ReorderBufferStreamTXN(rb, toptxn);
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
+ }
+
/* check the memory limits and evict something if needed */
ReorderBufferCheckMemoryLimit(rb);
}
@@ -700,7 +744,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
change->data.msg.message = palloc(message_size);
memcpy(change->data.msg.message, message, message_size);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -1862,8 +1906,8 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* disk.
*/
dlist_delete(&change->node);
- ReorderBufferToastAppendChunk(rb, txn, relation,
- change);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
}
change_done:
@@ -2456,7 +2500,7 @@ ReorderBufferAddSnapshot(ReorderBuffer *rb, TransactionId xid,
change->data.snapshot = snap;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2505,7 +2549,7 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
change->data.command_id = cid;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2528,6 +2572,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
Size sz;
ReorderBufferTXN *txn;
+ ReorderBufferTXN *toptxn = NULL;
Assert(change->txn);
@@ -2542,8 +2587,13 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
txn = change->txn;
/* if subxact, and streaming supported, use the toplevel instead */
- if (txn->toptxn && ReorderBufferCanStream(rb))
- txn = txn->toptxn;
+ if (ReorderBufferCanStream(rb))
+ {
+ if (txn->toptxn)
+ toptxn = txn->toptxn;
+ else
+ toptxn = txn;
+ }
sz = ReorderBufferChangeSize(change);
@@ -2551,12 +2601,20 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
txn->size += sz;
rb->size += sz;
+
+ /* Update the total size in the top transaction. */
+ if (toptxn)
+ toptxn->total_size += sz;
}
else
{
Assert((rb->size >= sz) && (txn->size >= sz));
txn->size -= sz;
rb->size -= sz;
+
+ /* Update the total size in the top transaction. */
+ if (toptxn)
+ toptxn->total_size -= sz;
}
Assert(txn->size <= rb->size);
@@ -2617,7 +2675,7 @@ ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
memcpy(change->data.inval.invalidations, msgs,
sizeof(SharedInvalidationMessage) * nmsgs);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -2804,15 +2862,16 @@ ReorderBufferLargestTopTXN(ReorderBuffer *rb)
txn = dlist_container(ReorderBufferTXN, node, iter.cur);
- /* if the current transaction is larger, remember it */
- if ((!largest) || (txn->size > largest->size))
- largest = txn;
+ /*
+ * if the current transaction is larger and doesn't have incomplete data
+ * remember it.
+ */
+ if (((!largest) || (txn->total_size > largest->total_size)) &&
+ ((txn->size > 0) && !rbtxn_has_toast_insert(txn) &&
+ !rbtxn_has_spec_insert(txn)))
+ largest = txn;
}
- Assert(largest);
- Assert(largest->size > 0);
- Assert(largest->size <= rb->size);
-
return largest;
}
@@ -2830,66 +2889,51 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
ReorderBufferTXN *txn;
- /* bail out if we haven't exceeded the memory limit */
- if (rb->size < logical_decoding_work_mem * 1024L)
- return;
-
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by streaming, if supported. Otherwise spill to disk.
- */
- if (ReorderBufferCanStream(rb))
+ /* Loop until we reach under the memory limit. */
+ while (rb->size >= logical_decoding_work_mem * 1024L)
{
/*
- * Pick the largest toplevel transaction and evict it from memory by
- * streaming the already decoded part.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTopTXN(rb);
+ if (ReorderBufferCanStream(rb) &&
+ (txn = ReorderBufferLargestTopTXN(rb)) != NULL)
+ {
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->total_size > 0);
+ Assert(rb->size >= txn->total_size);
- /* we know there has to be one, because the size is not zero */
- Assert(txn && !txn->toptxn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- ReorderBufferStreamTXN(rb, txn);
- }
- else
- {
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
- */
- txn = ReorderBufferLargestTXN(rb);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- /* we know there has to be one, because the size is not zero */
- Assert(txn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
- ReorderBufferSerializeTXN(rb, txn);
+ /*
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
+ */
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
}
- /*
- * After eviction, the transaction should have no entries in memory, and
- * should use 0 bytes for changes.
- *
- * XXX Checking the size is fine for both cases - spill to disk and
- * streaming. But for streaming we should really check nentries_mem for
- * all subtransactions too.
- */
- Assert(txn->size == 0);
- Assert(txn->nentries_mem == 0);
-
- /*
- * And furthermore, evicting the transaction should get us below the
- * memory limit again - it is not possible that we're still exceeding the
- * memory limit after evicting the transaction.
- *
- * This follows from the simple fact that the selected transaction is at
- * least as large as the most recent change (which caused us to go over
- * the memory limit). So by evicting it we're definitely back below the
- * memory limit.
- */
Assert(rb->size < logical_decoding_work_mem * 1024L);
}
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 95d18cd..aa17f7d 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -67,6 +67,7 @@
#define XLH_INSERT_LAST_IN_MULTI (1<<1)
#define XLH_INSERT_IS_SPECULATIVE (1<<2)
#define XLH_INSERT_CONTAINS_NEW_TUPLE (1<<3)
+#define XLH_INSERT_ON_TOAST_RELATION (1<<4)
/*
* xl_heap_update flag values, 8 bits are available.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 603f325..ba2ab71 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -172,6 +172,8 @@ typedef struct ReorderBufferChange
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
#define RBTXN_IS_STREAMED 0x0008
+#define RBTXN_HAS_TOAST_INSERT 0x0010
+#define RBTXN_HAS_SPEC_INSERT 0x0020
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -191,6 +193,17 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/* This transaction's changes has toast insert, without main table insert. */
+#define rbtxn_has_toast_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_TOAST_INSERT) != 0 \
+
+/*
+ * This transaction's changes has speculative insert, without speculative
+ * confirm.
+ */
+#define rbtxn_has_spec_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_SPEC_INSERT) != 0 \
+
/*
* Has this transaction been streamed to downstream?
*
@@ -199,10 +212,6 @@ typedef struct ReorderBufferChange
* which case we'd have nentries==0 for the toplevel one, which would say
* nothing about the streaming. So we maintain this flag, but only for the
* toplevel transaction.)
- *
- * Note: We never do both stream and serialize a transaction (we only spill
- * to disk when streaming is not supported by the plugin), so only one of
- * those two flags may be set at any given time.
*/
#define rbtxn_is_streamed(txn) \
( \
@@ -355,6 +364,9 @@ typedef struct ReorderBufferTXN
* Size of this transaction (changes currently in memory, in bytes).
*/
Size size;
+
+ /* Size of top-transaction including sub-transactions. */
+ Size total_size;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -545,7 +557,9 @@ void ReorderBufferReturnChange(ReorderBuffer *, ReorderBufferChange *);
Oid *ReorderBufferGetRelids(ReorderBuffer *, int nrelids);
void ReorderBufferReturnRelids(ReorderBuffer *, Oid *relids);
-void ReorderBufferQueueChange(ReorderBuffer *, TransactionId, XLogRecPtr lsn, ReorderBufferChange *);
+void ReorderBufferQueueChange(ReorderBuffer *, TransactionId,
+ XLogRecPtr lsn, ReorderBufferChange *,
+ bool incomplte_data);
void ReorderBufferQueueMessage(ReorderBuffer *, TransactionId, Snapshot snapshot, XLogRecPtr lsn,
bool transactional, const char *prefix,
Size message_size, const char *message);
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-14 15:44 ` Erik Rijkers <er@xs4all.nl>
2020-04-15 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 2 replies; 536+ messages in thread
From: Erik Rijkers @ 2020-04-14 15:44 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-04-14 12:10, Dilip Kumar wrote:
> v14-0001-Immediately-WAL-log-assignments.patch +
> v14-0002-Issue-individual-invalidations-with.patch +
> v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch+
> v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch+
> v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch +
> v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch+
> v14-0007-Track-statistics-for-streaming.patch +
> v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch +
> v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch +
> v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch
applied on top of 8128b0c (a few hours ago)
Hi,
I haven't followed this thread and maybe this instabilty is
known/expected; just thought I'd let you know.
When doing running a pgbench run over logical replication (cascading
down two replicas), I get this segmentation fault.
2020-04-14 17:27:28.135 CEST [8118] DETAIL: Streaming transactions
committing after 0/5FA2A38, reading WAL from 0/5FA2A00.
2020-04-14 17:27:28.135 CEST [8118] LOG: logical decoding found
consistent point at 0/5FA2A00
2020-04-14 17:27:28.135 CEST [8118] DETAIL: There are no running
transactions.
2020-04-14 17:27:28.138 CEST [8006] LOG: server process (PID 8118) was
terminated by signal 11: Segmentation fault
2020-04-14 17:27:28.138 CEST [8006] DETAIL: Failed process was running:
COMMIT
2020-04-14 17:27:28.138 CEST [8006] LOG: terminating any other active
server processes
2020-04-14 17:27:28.138 CEST [8163] WARNING: terminating connection
because of crash of another server process
2020-04-14 17:27:28.138 CEST [8163] DETAIL: The postmaster has
commanded this server process to roll back the current transaction and
exit, because another server process exited abnormally and possibly
corrupted shared memory.
2020-04-14 17:27:28.138 CEST [8163] HINT: In a moment you should be
able to reconnect to the database and repeat your command.
This error happens somewhat buried away in my test-stuff; I can dig it
out and make it into a repeatable test if you need it. (debian
stretch/gcc 9.3.0)
Erik Rijkers
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2020-04-15 04:44 ` Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-04-15 04:44 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Apr 14, 2020 at 9:14 PM Erik Rijkers <er@xs4all.nl> wrote:
>
> On 2020-04-14 12:10, Dilip Kumar wrote:
>
> > v14-0001-Immediately-WAL-log-assignments.patch +
> > v14-0002-Issue-individual-invalidations-with.patch +
> > v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch+
> > v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch+
> > v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch +
> > v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch+
> > v14-0007-Track-statistics-for-streaming.patch +
> > v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch +
> > v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch +
> > v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch
>
> applied on top of 8128b0c (a few hours ago)
>
> Hi,
>
> I haven't followed this thread and maybe this instabilty is
> known/expected; just thought I'd let you know.
>
> When doing running a pgbench run over logical replication (cascading
> down two replicas), I get this segmentation fault.
Thanks for the testing. Is it possible to share the call stack?
>
> 2020-04-14 17:27:28.135 CEST [8118] DETAIL: Streaming transactions
> committing after 0/5FA2A38, reading WAL from 0/5FA2A00.
> 2020-04-14 17:27:28.135 CEST [8118] LOG: logical decoding found
> consistent point at 0/5FA2A00
> 2020-04-14 17:27:28.135 CEST [8118] DETAIL: There are no running
> transactions.
> 2020-04-14 17:27:28.138 CEST [8006] LOG: server process (PID 8118) was
> terminated by signal 11: Segmentation fault
> 2020-04-14 17:27:28.138 CEST [8006] DETAIL: Failed process was running:
> COMMIT
> 2020-04-14 17:27:28.138 CEST [8006] LOG: terminating any other active
> server processes
> 2020-04-14 17:27:28.138 CEST [8163] WARNING: terminating connection
> because of crash of another server process
> 2020-04-14 17:27:28.138 CEST [8163] DETAIL: The postmaster has
> commanded this server process to roll back the current transaction and
> exit, because another server process exited abnormally and possibly
> corrupted shared memory.
> 2020-04-14 17:27:28.138 CEST [8163] HINT: In a moment you should be
> able to reconnect to the database and repeat your command.
>
>
> This error happens somewhat buried away in my test-stuff; I can dig it
> out and make it into a repeatable test if you need it. (debian
> stretch/gcc 9.3.0)
Yeah, that will be great.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2020-04-16 09:33 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-04-16 09:33 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Apr 14, 2020 at 9:14 PM Erik Rijkers <er@xs4all.nl> wrote:
>
> On 2020-04-14 12:10, Dilip Kumar wrote:
>
> > v14-0001-Immediately-WAL-log-assignments.patch +
> > v14-0002-Issue-individual-invalidations-with.patch +
> > v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch+
> > v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch+
> > v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch +
> > v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch+
> > v14-0007-Track-statistics-for-streaming.patch +
> > v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch +
> > v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch +
> > v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch
>
> applied on top of 8128b0c (a few hours ago)
Hi Erik,
While setting up the cascading replication I have hit one issue on
base code[1]. After fixing that I have got one crash with streaming
on patch. I am not sure whether you are facing any of these 2 issues
or any other issue. If your issue is not any of these then plese
share the callstack and steps to reproduce.
[1] https://www.postgresql.org/message-id/CAFiTN-u64S5bUiPL1q5kwpHNd0hRnf1OE-bzxNiOs5zo84i51w%40mail.gma...
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
bugfix_in_schema_sent.patch (application/octet-stream, 416B) download
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 59a09b9..811706a 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -999,7 +999,10 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
if (!found)
+ {
entry->schema_sent = false;
+ entry->streamed_txns = NULL;
+ }
return entry;
}
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-16 09:46 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2020-04-16 09:46 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-04-16 11:33, Dilip Kumar wrote:
> On Tue, Apr 14, 2020 at 9:14 PM Erik Rijkers <er@xs4all.nl> wrote:
>>
>> On 2020-04-14 12:10, Dilip Kumar wrote:
>>
>> > v14-0001-Immediately-WAL-log-assignments.patch +
>> > v14-0002-Issue-individual-invalidations-with.patch +
>> > v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch+
>> > v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch+
>> > v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch +
>> > v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch+
>> > v14-0007-Track-statistics-for-streaming.patch +
>> > v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch +
>> > v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch +
>> > v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch
>>
>> applied on top of 8128b0c (a few hours ago)
>
I've added your new patch
[bugfix_replica_identity_full_on_subscriber.patch]
on top of all those above but the crash (apparently the same crash) that
I had earlier still occurs (and pretty soon).
server process (PID 1721) was terminated by signal 11: Segmentation
fault
I'll try to isolate it better and get a stacktrace
> Hi Erik,
>
> While setting up the cascading replication I have hit one issue on
> base code[1]. After fixing that I have got one crash with streaming
> on patch. I am not sure whether you are facing any of these 2 issues
> or any other issue. If your issue is not any of these then plese
> share the callstack and steps to reproduce.
>
> [1]
> https://www.postgresql.org/message-id/CAFiTN-u64S5bUiPL1q5kwpHNd0hRnf1OE-bzxNiOs5zo84i51w%40mail.gma...
>
>
> --
> Regards,
> Dilip Kumar
> EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2020-04-18 09:07 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2020-04-18 09:07 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-04-16 11:46, Erik Rijkers wrote:
> On 2020-04-16 11:33, Dilip Kumar wrote:
>> On Tue, Apr 14, 2020 at 9:14 PM Erik Rijkers <er@xs4all.nl> wrote:
>>>
>>> On 2020-04-14 12:10, Dilip Kumar wrote:
>>>
>>> > v14-0001-Immediately-WAL-log-assignments.patch +
>>> > v14-0002-Issue-individual-invalidations-with.patch +
>>> > v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch+
>>> > v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch+
>>> > v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch +
>>> > v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch+
>>> > v14-0007-Track-statistics-for-streaming.patch +
>>> > v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch +
>>> > v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch +
>>> > v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch
>>>
>>> applied on top of 8128b0c (a few hours ago)
>>
>
> I've added your new patch
>
> [bugfix_replica_identity_full_on_subscriber.patch]
>
> on top of all those above but the crash (apparently the same crash)
> that I had earlier still occurs (and pretty soon).
>
> server process (PID 1721) was terminated by signal 11: Segmentation
> fault
>
> I'll try to isolate it better and get a stacktrace
>
>
>> Hi Erik,
>>
>> While setting up the cascading replication I have hit one issue on
>> base code[1]. After fixing that I have got one crash with streaming
>> on patch. I am not sure whether you are facing any of these 2 issues
>> or any other issue. If your issue is not any of these then plese
>> share the callstack and steps to reproduce.
I figured out a few things about this. Attached is a bash script
test.sh, to reproduce:
There is a variable CRASH_IT that determines whether the whole thing
will fail (with a segmentation fault) or not. As attached it has
CRASH_IT=0 and does not crash. When you change that to CRASH_IT=1, then
it will crash. It turns out that this just depends on a short wait
state (3 seconds, on my machine) between setting up de replication, and
the running of pgbench. It's possible that on very fast machines maybe
it does not occur; we've had such difference between hardware before.
This is a i5-3330S.
It deletes files so look it over before you run it. It may also depend
on some of my local set-up but I guess that should be easily fixed.
Can you let me know if you can reproduce the problem with this?
thanks,
Erik Rijkers
>>
>> [1]
>> https://www.postgresql.org/message-id/CAFiTN-u64S5bUiPL1q5kwpHNd0hRnf1OE-bzxNiOs5zo84i51w%40mail.gma...
>>
>>
>> --
>> Regards,
>> Dilip Kumar
>> EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2020-04-18 09:10 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2020-04-18 09:10 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-04-18 11:07, Erik Rijkers wrote:
>>> Hi Erik,
>>>
>>> While setting up the cascading replication I have hit one issue on
>>> base code[1]. After fixing that I have got one crash with streaming
>>> on patch. I am not sure whether you are facing any of these 2 issues
>>> or any other issue. If your issue is not any of these then plese
>>> share the callstack and steps to reproduce.
>
> I figured out a few things about this. Attached is a bash script
> test.sh, to reproduce:
And the attached file, test.sh. (sorry)
> There is a variable CRASH_IT that determines whether the whole thing
> will fail (with a segmentation fault) or not. As attached it has
> CRASH_IT=0 and does not crash. When you change that to CRASH_IT=1,
> then it will crash. It turns out that this just depends on a short
> wait state (3 seconds, on my machine) between setting up de
> replication, and the running of pgbench. It's possible that on very
> fast machines maybe it does not occur; we've had such difference
> between hardware before. This is a i5-3330S.
>
> It deletes files so look it over before you run it. It may also
> depend on some of my local set-up but I guess that should be easily
> fixed.
>
> Can you let me know if you can reproduce the problem with this?
>
> thanks,
>
> Erik Rijkers
>
>
>
>>>
>>> [1]
>>> https://www.postgresql.org/message-id/CAFiTN-u64S5bUiPL1q5kwpHNd0hRnf1OE-bzxNiOs5zo84i51w%40mail.gma...
>>>
>>>
>>> --
>>> Regards,
>>> Dilip Kumar
>>> EnterpriseDB: http://www.enterprisedb.com
Attachments:
test.sh (text/x-shellscript, 8.3K) download
#!/bin/bash
unset PGSERVICE PGSERVICEFILE PGDATA PGPORT PGDATABASE
# PGPASSFILE must be set and have the appropriate entries
env | grep -E '^PG'
PROJECT=large_logical
# PROJECT=HEAD
NUM_INSTANCES=3
BIN_DIR=$HOME/pg_stuff/pg_installations/pgsql.$PROJECT/bin
POSTGRES=$BIN_DIR/postgres
INITDB=$BIN_DIR/initdb
TMP_DIR=$HOME'/tmp/'$PROJECT
devel_file=${TMP_DIR}'/.devel'
BASE_PORT=6015 # ports 6015, 6016, 6017
# BASE_PORT=6515
port1=$(( $BASE_PORT + 0 ))
port2=$(( $port1 + 1 ))
port3=$(( $port1 + 2 ))
scale=1 dbname=postgres pubname=pub1 subname=sub1
if [[ ! -d $TMP_DIR ]]; then mkdir $TMP_DIR; fi
echo 's3kr1t' > $devel_file
max_wal_senders=10 # publication side
max_replication_slots=10 # publication side and subscription side
max_worker_processes=12 # subscription side
max_logical_replication_workers=10 # subscription side
max_sync_workers_per_subscription=4 # subscription side
for n in `seq 1 $NUM_INSTANCES`; do
port=$(( $BASE_PORT + $n -1 ))
data_dir=$TMP_DIR/pgsql.instance${n}/data
server_dir=$TMP_DIR/pgsql.instance${n}
$INITDB --pgdata=$data_dir --encoding=UTF8 --auth=scram-sha-256 --pwfile=$devel_file &> initdb.$port.log
rc=$?
if [[ $rc -ne 0 ]]; then
echo "-- initdb $?"
fi
( $POSTGRES -D $data_dir -p $port \
--wal_level=logical \
--max_replication_slots=$max_replication_slots \
--max_worker_processes=$max_worker_processes \
--max_logical_replication_workers=$max_logical_replication_workers \
--max_wal_senders=$max_wal_senders \
--max_sync_workers_per_subscription=$max_sync_workers_per_subscription \
--logging_collector=on \
--log_directory=${server_dir} \
--log_filename=logfile.${port} \
--log_replication_commands=on \
--autovacuum=off & )
# --logical_work_mem=128MB & )
# pg_isready -d $dbname --timeout=60 -p $port
done
#sleep $NUM_INSTANCES
#sleep $NUM_INSTANCES
#pg_isready -d $dbname -qp 6015 --timeout=60
#pg_isready -d $dbname -qp 6016 --timeout=60
num_loop=$(( $NUM_INSTANCES - 1 ))
$BIN_DIR/pgbench --port=$BASE_PORT --quiet --initialize --scale=$scale $dbname
echo "alter table pgbench_history add column hid serial primary key" | $BIN_DIR/psql -d $dbname -p $BASE_PORT -X
#pg_isready -d $dbname -qp 6015 --timeout=60
#pg_isready -d $dbname -qp 6016 --timeout=60
for n in `seq 1 $num_loop`; do
target_port=$(( $BASE_PORT + $n ))
pg_dump -Fc -p $BASE_PORT \
--exclude-table-data=pgbench_history --exclude-table-data=pgbench_accounts \
--exclude-table-data=pgbench_branches --exclude-table-data=pgbench_tellers \
-tpgbench_history -tpgbench_accounts \
-tpgbench_branches -tpgbench_tellers \
$dbname | pg_restore -1 -p $target_port -d $dbname
done
for n in `seq 1 $num_loop`; do
pubport=$(( $BASE_PORT + $n - 1 ))
subport=$(( $BASE_PORT + $n ))
appname='casc:'${subport}'<'${pubport}
echo "create publication $pubname for all tables" | psql -d $dbname -p $pubport -X
echo "create subscription $subname
connection 'port=${pubport} dbname=${dbname} application_name=${appname}'
publication $pubname with (enabled=false, slot_name=${subname}_${subport});" | psql -d $dbname -p $subport -X
echo "alter subscription $subname enable; " | psql -d $dbname -p $subport -X
done
CRASH_IT=0 # 0: no crash; 1: crash
if [[ $CRASH_IT -ne 1 ]]
then
echo "-- sleep 3"
sleep 3 # this wait will avoid the crash
fi
echo "------------------------ (doing a pgbench run for 3 seconds)"
echo "-- $BIN_DIR/pgbench -c 32 -T 3 -p $BASE_PORT $dbname -- scale $scale"
$BIN_DIR/pgbench -c 32 -T 3 -p $BASE_PORT $dbname
echo "------------------------ (and back from pgbench run)"
c_a1=$( echo "select count(*) from pgbench_accounts"|psql -d$dbname -qtAX -p$port1)
c_b1=$( echo "select count(*) from pgbench_branches"|psql -d$dbname -qtAX -p$port1)
c_t1=$( echo "select count(*) from pgbench_tellers "|psql -d$dbname -qtAX -p$port1)
c_h1=$( echo "select count(*) from pgbench_history "|psql -d$dbname -qtAX -p$port1)
mda1=$( echo "select aid,bid,abalance,filler from pgbench_accounts order by aid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
mdb1=$( echo "select bid,bbalance,filler from pgbench_branches order by bid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
mdt1=$( echo "select tid,bid,tbalance,filler from pgbench_tellers order by tid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
mdh1=$( echo "select hid,bid,aid,delta,mtime,filler,hid from pgbench_history order by hid"|psql -d $dbname -qtAX -p$port1|md5sum|cut -b 1-7 )
md5_1=$(echo "$mda1 $mdb1 $mdt1 $mdh1" | md5sum | cut -b 1-7 )
printf "a,b,t,h $port1 %6d %6d %6d %6d $mda1 $mdb1 $mdt1 $mdh1 $md5_1\n" $c_a1 $c_b1 $c_t1 $c_h1
ver1=$( echo "select substring(version(),1,70)" | psql -d $dbname -qtAXp $port1 )
ver2=$( echo "select substring(version(),1,70)" | psql -d $dbname -qtAXp $port2 )
if [[ $NUM_INSTANCES -gt 2 ]]; then
ver3=$( echo "select substring(version(),1,70)" | psql -d $dbname -qtAXp $port3 )
fi
echo
rc=0
while [[ $rc -eq 0 ]]
do
# echo "select column_name from information_schema.columns where table_schema = 'public' and table_name = 'pgbench_accounts' order by ordinal_position"|psql -d$dbname -qtAXp$port2
mda2=$(echo "select aid,bid,abalance,filler from pgbench_accounts order by aid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
mdb2=$(echo "select bid,bbalance,filler from pgbench_branches order by bid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
mdt2=$(echo "select tid,bid,tbalance,filler from pgbench_tellers order by tid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
mdh2=$(echo "select hid,bid,aid,delta,mtime,filler,hid from pgbench_history order by hid"|psql -d$dbname -qtAXp$port2|md5sum|cut -b 1-7)
c_a2=$(echo "select count(*) from pgbench_accounts"|psql -d$dbname -qtAX -p$port2)
c_b2=$(echo "select count(*) from pgbench_branches"|psql -d$dbname -qtAX -p$port2)
c_t2=$(echo "select count(*) from pgbench_tellers "|psql -d$dbname -qtAX -p$port2)
c_h2=$(echo "select count(*) from pgbench_history "|psql -d$dbname -qtAX -p$port2)
md5_2=$(echo "$mda2 $mdb2 $mdt2 $mdh2" | md5sum | cut -b 1-7 )
if [[ $NUM_INSTANCES -gt 2 ]]; then
mda3=$(echo "select aid,bid,abalance,filler from pgbench_accounts order by aid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
mdb3=$(echo "select bid,bbalance,filler from pgbench_branches order by bid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
mdt3=$(echo "select tid,bid,tbalance,filler from pgbench_tellers order by tid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
mdh3=$(echo "select hid,bid,aid,delta,mtime,filler,hid from pgbench_history order by hid"|psql -d$dbname -qtAXp$port3|md5sum|cut -b 1-7)
c_a3=$(echo "select count(*) from pgbench_accounts"|psql -d$dbname -qtAX -p$port3)
c_b3=$(echo "select count(*) from pgbench_branches"|psql -d$dbname -qtAX -p$port3)
c_t3=$(echo "select count(*) from pgbench_tellers "|psql -d$dbname -qtAX -p$port3)
c_h3=$(echo "select count(*) from pgbench_history "|psql -d$dbname -qtAX -p$port3)
md5_3=$(echo "$mda3 $mdb3 $mdt3 $mdh3" | md5sum | cut -b 1-7 )
fi
echo "-- $POSTGRES"
printf "a,b,t,h $port1 %7d %6d %6d %6d $mda1 $mdb1 $mdt1 $mdh1 $md5_1 " $c_a1 $c_b1 $c_t1 $c_h1; echo "$ver1"
printf "a,b,t,h $port2 %7d %6d %6d %6d $mda2 $mdb2 $mdt2 $mdh2 $md5_2 " $c_a2 $c_b2 $c_t2 $c_h2; echo "$ver2"
if [[ $NUM_INSTANCES -gt 2 ]]; then
printf "a,b,t,h $port3 %7d %6d %6d %6d $mda3 $mdb3 $mdt3 $mdh3 $md5_3 " $c_a3 $c_b3 $c_t3 $c_h3; echo "$ver3"
fi
if [[ $NUM_INSTANCES -eq 2 ]]; then
if [[ "$md5_1" == "$md5_2" ]] ; then echo "OK - done."; break; fi
elif [[ $NUM_INSTANCES -eq 3 ]]; then
if [[ "$md5_1" == "$md5_3" ]] ; then echo "OK - done."; break; fi
fi
sleep 1
rc=$?
done
for n in `seq 1 $NUM_INSTANCES`; do
port=$(( $BASE_PORT + $n -1 ))
inst_dir=$TMP_DIR/pgsql.instance${n}
data_dir=$TMP_DIR/pgsql.instance${n}/data
echo "-- grep -Ei fault $TMP_DIR/pgsql.instance${n}/logf*"
grep -Ei fault $TMP_DIR/pgsql.instance${n}/logf*
sleep 2
$BIN_DIR/pg_ctl stop -w -D $data_dir
if [[ 1 -eq 1 ]]; then
echo "-- removing the lot: rm -rf $inst_dir"
rm -rf $inst_dir
fi
done
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2020-04-18 12:42 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 14:01 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-21 12:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Erik Rijkers @ 2020-04-18 12:42 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-04-18 11:10, Erik Rijkers wrote:
> On 2020-04-18 11:07, Erik Rijkers wrote:
>>>> Hi Erik,
>>>>
>>>> While setting up the cascading replication I have hit one issue on
>>>> base code[1]. After fixing that I have got one crash with streaming
>>>> on patch. I am not sure whether you are facing any of these 2
>>>> issues
>>>> or any other issue. If your issue is not any of these then plese
>>>> share the callstack and steps to reproduce.
>>
>> I figured out a few things about this. Attached is a bash script
>> test.sh, to reproduce:
>
> And the attached file, test.sh. (sorry)
It turns out I must have been mistaken somewhere. I probably missed
bugfix_in_schema_sent.patch)
I have just now rebuilt all the instances on top of master with these
patches:
> [v14-0001-Immediately-WAL-log-assignments.patch]
> [v14-0002-Issue-individual-invalidations-with.patch]
> [v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch]
> [v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch]
> [v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch]
> [v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch]
> [v14-0007-Track-statistics-for-streaming.patch]
> [v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch]
> [v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch]
> [v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch]
> [bugfix_in_schema_sent.patch]
(by the way: this build's regression tests 'ddl', 'toast', and
'spill' fail)
I seem now able to run all my test programs on these instances without
errors.
Sorry, I seem to have raised a false alarm (although there was initially
certainly a problem).
Erik Rijkers
>> There is a variable CRASH_IT that determines whether the whole thing
>> will fail (with a segmentation fault) or not. As attached it has
>> CRASH_IT=0 and does not crash. When you change that to CRASH_IT=1,
>> then it will crash. It turns out that this just depends on a short
>> wait state (3 seconds, on my machine) between setting up de
>> replication, and the running of pgbench. It's possible that on very
>> fast machines maybe it does not occur; we've had such difference
>> between hardware before. This is a i5-3330S.
>>
>> It deletes files so look it over before you run it. It may also
>> depend on some of my local set-up but I guess that should be easily
>> fixed.
>>
>> Can you let me know if you can reproduce the problem with this?
>>
>> thanks,
>>
>> Erik Rijkers
>>
>>
>>
>>>>
>>>> [1]
>>>> https://www.postgresql.org/message-id/CAFiTN-u64S5bUiPL1q5kwpHNd0hRnf1OE-bzxNiOs5zo84i51w%40mail.gma...
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Dilip Kumar
>>>> EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2020-04-18 14:01 ` Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-04-18 14:01 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Apr 18, 2020 at 6:12 PM Erik Rijkers <er@xs4all.nl> wrote:
>
> On 2020-04-18 11:10, Erik Rijkers wrote:
> > On 2020-04-18 11:07, Erik Rijkers wrote:
> >>>> Hi Erik,
> >>>>
> >>>> While setting up the cascading replication I have hit one issue on
> >>>> base code[1]. After fixing that I have got one crash with streaming
> >>>> on patch. I am not sure whether you are facing any of these 2
> >>>> issues
> >>>> or any other issue. If your issue is not any of these then plese
> >>>> share the callstack and steps to reproduce.
> >>
> >> I figured out a few things about this. Attached is a bash script
> >> test.sh, to reproduce:
> >
> > And the attached file, test.sh. (sorry)
>
> It turns out I must have been mistaken somewhere. I probably missed
> bugfix_in_schema_sent.patch)
>
> I have just now rebuilt all the instances on top of master with these
> patches:
>
> > [v14-0001-Immediately-WAL-log-assignments.patch]
> > [v14-0002-Issue-individual-invalidations-with.patch]
> > [v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch]
> > [v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch]
> > [v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch]
> > [v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch]
> > [v14-0007-Track-statistics-for-streaming.patch]
> > [v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch]
> > [v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch]
> > [v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch]
> > [bugfix_in_schema_sent.patch]
>
> (by the way: this build's regression tests 'ddl', 'toast', and
> 'spill' fail)
>
> I seem now able to run all my test programs on these instances without
> errors.
>
> Sorry, I seem to have raised a false alarm (although there was initially
> certainly a problem).
No problem, Thanks for confirming.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2020-04-21 12:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 14:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-04-21 12:00 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Sat, Apr 18, 2020 at 6:12 PM Erik Rijkers <er@xs4all.nl> wrote:
>
> On 2020-04-18 11:10, Erik Rijkers wrote:
> > On 2020-04-18 11:07, Erik Rijkers wrote:
> >>>> Hi Erik,
> >>>>
> >>>> While setting up the cascading replication I have hit one issue on
> >>>> base code[1]. After fixing that I have got one crash with streaming
> >>>> on patch. I am not sure whether you are facing any of these 2
> >>>> issues
> >>>> or any other issue. If your issue is not any of these then plese
> >>>> share the callstack and steps to reproduce.
> >>
> >> I figured out a few things about this. Attached is a bash script
> >> test.sh, to reproduce:
> >
> > And the attached file, test.sh. (sorry)
>
> It turns out I must have been mistaken somewhere. I probably missed
> bugfix_in_schema_sent.patch)
>
> I have just now rebuilt all the instances on top of master with these
> patches:
>
> > [v14-0001-Immediately-WAL-log-assignments.patch]
> > [v14-0002-Issue-individual-invalidations-with.patch]
> > [v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch]
> > [v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch]
> > [v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch]
> > [v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch]
> > [v14-0007-Track-statistics-for-streaming.patch]
> > [v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch]
> > [v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch]
> > [v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch]
> > [bugfix_in_schema_sent.patch]
>
> (by the way: this build's regression tests 'ddl', 'toast', and
> 'spill' fail)
Yeah, this is a. known issue, actually, while streaming the
transaction the output message is changed. I have a plan to work on
this part.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-21 12:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-22 14:49 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 16:01 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-04-22 14:49 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Apr 21, 2020 at 5:30 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Sat, Apr 18, 2020 at 6:12 PM Erik Rijkers <er@xs4all.nl> wrote:
> >
> > On 2020-04-18 11:10, Erik Rijkers wrote:
> > > On 2020-04-18 11:07, Erik Rijkers wrote:
> > >>>> Hi Erik,
> > >>>>
> > >>>> While setting up the cascading replication I have hit one issue on
> > >>>> base code[1]. After fixing that I have got one crash with streaming
> > >>>> on patch. I am not sure whether you are facing any of these 2
> > >>>> issues
> > >>>> or any other issue. If your issue is not any of these then plese
> > >>>> share the callstack and steps to reproduce.
> > >>
> > >> I figured out a few things about this. Attached is a bash script
> > >> test.sh, to reproduce:
> > >
> > > And the attached file, test.sh. (sorry)
> >
> > It turns out I must have been mistaken somewhere. I probably missed
> > bugfix_in_schema_sent.patch)
> >
> > I have just now rebuilt all the instances on top of master with these
> > patches:
> >
> > > [v14-0001-Immediately-WAL-log-assignments.patch]
> > > [v14-0002-Issue-individual-invalidations-with.patch]
> > > [v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch]
> > > [v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch]
> > > [v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch]
> > > [v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch]
> > > [v14-0007-Track-statistics-for-streaming.patch]
> > > [v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch]
> > > [v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch]
> > > [v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch]
> > > [bugfix_in_schema_sent.patch]
> >
> > (by the way: this build's regression tests 'ddl', 'toast', and
> > 'spill' fail)
>
> Yeah, this is a. known issue, actually, while streaming the
> transaction the output message is changed. I have a plan to work on
> this part.
I have fixed this part. Basically, now, I have created a separate
function to get the streaming changes
'pg_logical_slot_get_streaming_changes'. So the default function
pg_logical_slot_get_changes will work as it is and test decoding test
cases will not fail.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v15-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.5K) download
From e856c586bb131e1047471ecfddc6b0f118f1b0f3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 20 Mar 2020 15:03:01 +0530
Subject: [PATCH v15 01/11] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 46 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 +++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 ++++++++++++++-------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 99 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 3984dd3..c2604bb 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -191,6 +191,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -223,6 +224,7 @@ typedef struct SerializedTransactionState
static TransactionStateData TopTransactionStateData = {
.state = TRANS_DEFAULT,
.blockState = TBLOCK_DEFAULT,
+ .assigned = false,
};
/*
@@ -5118,6 +5120,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6020,3 +6023,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 4259309..3c49954 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -89,11 +89,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfXLogTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfXLogTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -195,6 +197,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -398,7 +404,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -748,6 +754,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ XLogSetRecordFlags(XLOG_INCLUDE_XID);
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 79ff976..7b5257f 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1189,6 +1189,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1227,6 +1228,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index c2e5e3a..122c581 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel xid is valid, we need to assign the subxact to the
+ * toplevel xact. We need to do this for all records, hence we do it before
+ * the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ txid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrId) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(XLogRecGetTopXid(r)))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index f60ed2d..6d439d0 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -229,6 +229,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 4582196..756f6df 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -150,6 +150,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -285,6 +287,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v15-0002-Issue-individual-invalidations-with-wal_level-lo.patch (application/octet-stream, 16.7K) download
From 36577ba865b02113645c76c92b214b606a634727 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Tue, 14 Apr 2020 11:11:37 +0530
Subject: [PATCH v15 02/11] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
---
src/backend/access/rmgrdesc/xactdesc.c | 40 +++++++++
src/backend/access/transam/xact.c | 7 ++
src/backend/replication/logical/decode.c | 16 ++++
src/backend/replication/logical/reorderbuffer.c | 104 +++++++++++++++++++++---
src/backend/utils/cache/inval.c | 49 +++++++++++
src/include/access/xact.h | 13 ++-
src/include/replication/reorderbuffer.h | 11 +++
7 files changed, 226 insertions(+), 14 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index fbc5942..17c06f7 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,9 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -396,6 +399,12 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs);
+ }
}
const char *
@@ -423,7 +432,38 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs)
+{
+ int i;
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index c2604bb..8e6b1a6 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6020,6 +6020,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 122c581..69c1f45 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,22 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ Assert(TransactionIdIsValid(xid));
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->nmsgs, invals->msgs);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 4594cf9..0d5bb73 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -220,7 +220,7 @@ static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
-static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs);
/*
* ---------------------------------------
@@ -455,6 +455,11 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
pfree(change->data.msg.message);
change->data.msg.message = NULL;
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ if (change->data.inval.invalidations)
+ pfree(change->data.inval.invalidations);
+ change->data.inval.invalidations = NULL;
+ break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
if (change->data.snapshot)
{
@@ -1814,17 +1819,24 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation messages locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ ReorderBufferExecuteInvalidations(
+ change->data.inval.ninvalidations,
+ change->data.inval.invalidations);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -1866,7 +1878,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -1892,7 +1905,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2204,6 +2218,33 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, int nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ MemoryContextAlloc(rb->context,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2234,12 +2275,12 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
* in the changestream but we don't know which those are.
*/
static void
-ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs)
{
int i;
- for (i = 0; i < txn->ninvalidations; i++)
- LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+ for (i = 0; i < nmsgs; i++)
+ LocalExecuteInvalidationMessage(&msgs[i]);
}
/*
@@ -2591,6 +2632,24 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ {
+ char *data;
+ Size inval_size = sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+
+ sz += inval_size;
+
+ ReorderBufferSerializeReserve(rb, sz);
+ data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
+
+ /* might have been reallocated above */
+ ondisk = (ReorderBufferDiskChange *) rb->outbuf;
+ memcpy(data, change->data.inval.invalidations, inval_size);
+ data += inval_size;
+
+ break;
+ }
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot snap;
@@ -2736,6 +2795,11 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ sz += sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot snap;
@@ -3004,6 +3068,20 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ {
+ Size inval_size = sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+
+ change->data.inval.invalidations =
+ MemoryContextAlloc(rb->context,
+ change->data.msg.message_size);
+ /* read the message */
+ memcpy(change->data.inval.invalidations, data, inval_size);
+ data += inval_size;
+
+ break;
+ }
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot oldsnap;
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..cba5b6c 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write invalidations into WAL at each command end to
+ * support the decoding of the in-progress transaction. As of now it was
+ * enough to log invalidation only at commit because we are only decoding the
+ * transaction at the commit time. We only need to log the catalog cache and
+ * relcache invalidation. There can not be any active MVCC scan in logical
+ * decoding so we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,8 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(void);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -1092,6 +1101,9 @@ CommandEndInvalidationMessages(void)
if (transInvalInfo == NULL)
return;
+ if (XLogLogicalInfoActive())
+ LogLogicalInvalidations();
+
ProcessInvalidationMessages(&transInvalInfo->CurrentCmdInvalidMsgs,
LocalExecuteInvalidationMessage);
AppendInvalidationMessages(&transInvalInfo->PriorCmdInvalidMsgs,
@@ -1501,3 +1513,40 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations()
+{
+ xl_xact_invalidations xlrec;
+ SharedInvalidationMessage *invalMessages;
+ int nmsgs = 0;
+
+ if (transInvalInfo->CurrentCmdInvalidMsgs.cclist)
+ {
+ ProcessInvalidationMessagesMulti(&transInvalInfo->CurrentCmdInvalidMsgs,
+ MakeSharedInvalidMessagesArray);
+ invalMessages = SharedInvalidMessagesArray;
+ nmsgs = numSharedInvalidMessagesArray;
+ SharedInvalidMessagesArray = NULL;
+ numSharedInvalidMessagesArray = 0;
+ }
+
+ if (nmsgs > 0)
+ {
+ /* prepare record */
+ memset(&xlrec, 0, MinSizeOfXactInvalidations);
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) invalMessages,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+
+ pfree(invalMessages);
+ }
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..b822c5e 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,17 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ */
+typedef struct xl_xact_invalidations
+{
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 626ecf4..af35287 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,14 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ uint32 ninvalidations; /* Number of messages */
+ SharedInvalidationMessage *invalidations; /* invalidation
+ * message */
+ } inval;
} data;
/*
@@ -459,6 +468,8 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ int nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v15-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch (application/octet-stream, 12.3K) download
From 1ef5f6fbc9525a24f83e43cdeea9f46a13848cbd Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Apr 2020 10:55:19 +0530
Subject: [PATCH v15 04/11] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 38 +++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++++--
src/include/utils/snapmgr.h | 4 ++-
6 files changed, 113 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 65244b1..b59a6c8 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>pg_current_xact_id()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0d4ed60..84884a4 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1288,6 +1288,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
if (scan->rs_base.rs_flags & SO_ALLOW_PAGEMODE)
@@ -1383,6 +1392,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1497,6 +1514,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1646,6 +1671,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5451,6 +5484,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_finish_speculative with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_finish_speculative call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index dfba5ae..97a1075 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -433,6 +434,25 @@ systable_beginscan(Relation heapRelation,
}
/*
+ * HandleConcurrentAbort - Handle concurrent abort of the CheckXidAlive.
+ *
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we error
+ * out. We can't directly use TransactionIdDidAbort as after crash such
+ * transaction might not have been marked as aborted. See detailed comments at
+ * snapmgr.c where the variable is declared.
+ */
+static inline void
+HandleConcurrentAbort()
+{
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+}
+
+/*
* systable_getnext --- get next tuple in a heap-or-index scan
*
* Returns NULL if no more tuples available.
@@ -481,6 +501,12 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * Handle the concurrent abort while fetching the catalog tuple during
+ * logical streaming of a transaction.
+ */
+ HandleConcurrentAbort();
+
return htup;
}
@@ -517,6 +543,12 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * Handle the concurrent abort while fetching the catalog tuple during
+ * logical streaming of a transaction.
+ */
+ HandleConcurrentAbort();
+
return result;
}
@@ -643,6 +675,12 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * Handle the concurrent abort while fetching the catalog tuple during
+ * logical streaming of a transaction.
+ */
+ HandleConcurrentAbort();
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0d5bb73..2302875 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -696,7 +696,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1547,7 +1547,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1798,7 +1798,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1818,7 +1818,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v15-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From a65eda69a3bd6810223ea321d24f8080451e3351 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v15 03/11] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 93c9488..64f651f 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -540,3 +569,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index bad3bfe..65244b1 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 5adf253..497d8a9 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -862,6 +910,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index af35287..e102840 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -354,6 +354,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -393,6 +439,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v15-0005-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 37.9K) download
From e10c22b9424d66d5bff25faa9c24c2e41c110276 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v15 05/11] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 713 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 36 ++
3 files changed, 696 insertions(+), 91 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 2302875..801fdc5 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static inline bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -773,6 +786,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -865,6 +910,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -988,7 +1036,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1024,6 +1072,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1038,6 +1089,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1316,6 +1370,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1341,8 +1404,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1350,9 +1498,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1491,63 +1636,75 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+/*
+ * Set xid for concurrent abort check
+ *
+ * While streaming an in-progress transaction there is a possibility that the
+ * (sub)transaction might get aborted concurrently. In such case if the
+ * (sub)transaction has catalog update then we might decode the tuple using
+ * wrong catalog version. So for detecting the concurrent abort we set
+ * CheckXidAlive to the current (sub)transaction's xid for which this change
+ * belongs to. And, during catalog scan we can check the status of the xid and
+ * if it is aborted we will report a specific error that we can ignore. We
+ * might have already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the abort we will
+ * stream abort message to truncate the changes in the subscriber.
+ */
+static inline void
+SetupCheckXidLive(TransactionId xid)
+{
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
+ if (!TransactionIdDidCommit(xid))
+ CheckXidAlive = xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
- snapshot_now = txn->base_snapshot;
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
- /* build data to be able to lookup the CommandIds of catalog tuples */
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1563,15 +1720,20 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
+ XLogRecPtr prev_lsn = InvalidXLogRecPtr;
ReorderBufferChange *change;
ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1579,6 +1741,19 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* Set the xid for concurrent abort check. */
+ if (streaming)
+ SetupCheckXidLive(change->txn->xid);
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1588,8 +1763,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1655,7 +1828,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1676,8 +1857,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1695,7 +1874,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1753,7 +1932,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1762,10 +1949,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1796,9 +1989,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1818,7 +2011,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1858,14 +2052,46 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ if (!XLogRecPtrIsInvalid(prev_lsn))
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1884,14 +2110,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,18 +2145,125 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ Snapshot snapshot_now;
+ CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1946,6 +2287,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * If the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2015,6 +2363,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * If the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2150,8 +2505,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2159,6 +2523,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2170,19 +2535,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2211,6 +2585,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2295,6 +2670,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2399,6 +2781,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2418,15 +2832,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- ReorderBufferSerializeTXN(rb, txn);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2746,6 +3191,102 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static inline bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after the last
+ * streaming run, we might have got some new sub-transactions. So we
+ * need to add them to the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ Assert(txn->snapshot_now->copied);
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e102840..6d65986 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -171,6 +171,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -190,6 +191,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -225,6 +244,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -255,6 +284,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v15-0006-Add-support-for-streaming-to-built-in-replicatio.patch (application/octet-stream, 90.9K) download
From 5918775be74d7b7b99091d445c0fbf5d39dc8430 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Thu, 16 Apr 2020 01:55:22 -0700
Subject: [PATCH v15 06/11] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 45 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/launcher.c | 1 -
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 140 ++-
src/backend/replication/logical/worker.c | 1026 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 315 +++++-
src/backend/replication/slotfuncs.c | 6 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2033 insertions(+), 43 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 1b8bead..95b7c24 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -164,8 +164,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..3349cc4 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731..f28482f 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 7f15667..65b6b76 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +91,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -175,6 +178,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -318,6 +331,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -334,7 +349,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -412,6 +427,13 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +691,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -697,6 +722,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +740,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +779,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +816,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 50eea2e..4ef4fd4 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4133,6 +4133,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9..5257ab0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -408,6 +408,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..8156a42 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,7 +14,6 @@
*
*-------------------------------------------------------------------------
*/
-
#include "postgres.h"
#include "access/heapam.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 497d8a9..dfc681d 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1148,7 +1148,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1193,7 +1193,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = txn->final_lsn;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd..5242ac0 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,10 +139,15 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -178,8 +183,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -187,6 +192,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -248,7 +257,8 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -256,6 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -296,6 +310,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -305,6 +320,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -347,12 +366,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -397,7 +420,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -405,6 +428,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -685,3 +712,104 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendbyte(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgbyte(in) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a752a12..3dc5f83 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in tmp file directory by default, and the filenames
+ * include both the XID of the toplevel transaction and OID of the subscription.
+ * This is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -33,6 +54,7 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -64,6 +86,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -71,6 +94,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -110,12 +134,57 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
static void apply_handle_insert_internal(ResultRelInfo *relinfo,
EState *estate, TupleTableSlot *remoteslot);
static void apply_handle_update_internal(ResultRelInfo *relinfo,
@@ -187,6 +256,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -553,6 +658,318 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* XXX optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ CloseTransientFile(fd);
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid, false);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -565,6 +982,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -580,6 +1000,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -616,6 +1039,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -731,6 +1157,9 @@ apply_handle_update(StringInfo s)
RangeTblEntry *target_rte;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -873,6 +1302,9 @@ apply_handle_delete(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -1243,6 +1675,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1381,6 +1816,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1478,6 +1929,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i], true);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1493,6 +1960,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1941,6 +2411,561 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ CloseTransientFile(fd);
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ CloseTransientFile(fd);
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * missing_ok - don't report error for missing file is the flag is passed true.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid, true);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialized in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -2106,6 +3131,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 77b85fc..59a09b9 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -46,29 +46,54 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferChange *change);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-static void send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx);
+static void send_relation_and_attrs(Relation relation, TransactionId xid,
+ LogicalDecodingContext *ctx);
/*
* Entry in the map used to remember which relation schemas we sent.
*
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ *
* For partitions, 'pubactions' considers not only the table's own
* publications, but also those of all of its ancestors.
*/
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
-
+ TransactionId xid; /* transaction that created the record */
/*
* Did we send the schema? If ancestor relid is set, its schema must also
* have been sent for this to be true.
*/
bool schema_sent;
+ List *streamed_txns; /* streamed toplevel transactions with this
+ * schema */
bool replicate_valid;
PublicationActions pubactions;
@@ -94,11 +119,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -114,15 +145,24 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -168,6 +208,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -180,6 +237,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -202,7 +260,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -222,6 +281,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -290,9 +370,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (schema_sent)
return;
/* If needed, send the ancestor's schema first. */
@@ -308,19 +427,25 @@ maybe_send_schema(LogicalDecodingContext *ctx,
relentry->map = convert_tuples_by_name(CreateTupleDescCopy(indesc),
CreateTupleDescCopy(outdesc));
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, ctx);
- relentry->schema_sent = true;
+ send_relation_and_attrs(relation, xid, ctx);
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
/*
* Sends a relation
*/
static void
-send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx)
+send_relation_and_attrs(Relation relation, TransactionId xid,
+ LogicalDecodingContext *ctx)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -344,17 +469,19 @@ send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx)
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -363,6 +490,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -391,7 +522,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
@@ -411,7 +542,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, xid, relation, tuple);
OutputPluginWrite(ctx, true);
break;
}
@@ -435,7 +566,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple, newtuple);
OutputPluginWrite(ctx, true);
break;
}
@@ -455,7 +586,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, xid, relation, oldtuple);
OutputPluginWrite(ctx, true);
}
else
@@ -480,6 +611,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -508,13 +643,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -588,6 +724,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -624,6 +845,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*
* This looks up publications that the given relation is directly or
@@ -756,6 +1005,36 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -790,7 +1069,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index f776de3..9121420 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -156,6 +156,12 @@ create_logical_replication_slot(char *name, char *plugin,
NULL);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* If caller needs us to determine the decoding start point, do so now.
* This might take a while.
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 0e93322..eacea12 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1004,6 +1004,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..617b909 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b8041d9..3b3e1fd 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -981,7 +981,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561..89158ed 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -86,25 +90,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index f1aa6e9..70d39f8 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v15-0009-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From a05c493fe4bd2ee009525b4e5050c4657c5bd35b Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v15 09/11] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v15-0008-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From fddff6b6ba5f078c12324642e0ab6ac647cc4426 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v15 08/11] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 6da7f71..086d0c7 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -70,7 +70,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v15-0010-Bugfix-handling-of-incomplete-toast-tuple.patch (application/octet-stream, 15.3K) download
From b3a19c455130853bc4efb12bd68933a5224cad86 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 28 Feb 2020 11:07:46 +0530
Subject: [PATCH v15 10/11] Bugfix handling of incomplete toast tuple
---
src/backend/access/heap/heapam.c | 3 +
src/backend/replication/logical/decode.c | 17 ++-
src/backend/replication/logical/reorderbuffer.c | 182 +++++++++++++++---------
src/include/access/heapam_xlog.h | 1 +
src/include/replication/reorderbuffer.h | 24 +++-
5 files changed, 147 insertions(+), 80 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 84884a4..18b0bad 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1978,6 +1978,9 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
{
xlrec.flags |= XLH_INSERT_CONTAINS_NEW_TUPLE;
bufflags |= REGBUF_KEEP_DATA;
+
+ if (IsToastRelation(relation))
+ xlrec.flags |= XLH_INSERT_ON_TOAST_RELATION;
}
XLogBeginInsert();
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 69c1f45..c841687 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -727,7 +727,9 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change,
+ xlrec->flags & XLH_INSERT_ON_TOAST_RELATION);
}
/*
@@ -794,7 +796,8 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -851,7 +854,8 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -887,7 +891,7 @@ DecodeTruncate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
memcpy(change->data.truncate.relids, xlrec->relids,
xlrec->nrelids * sizeof(Oid));
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
}
/*
@@ -987,7 +991,7 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = false;
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
/* move to the next xl_multi_insert_tuple entry */
data += datalen;
@@ -1025,7 +1029,8 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 845d820..f299c64 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -654,11 +654,14 @@ ReorderBufferTXNByXid(ReorderBuffer *rb, TransactionId xid, bool create,
*/
void
ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
- ReorderBufferChange *change)
+ ReorderBufferChange *change, bool toast_insert)
{
- ReorderBufferTXN *txn;
+ ReorderBufferTXN *txn, *toptxn;
+ bool can_stream = false;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ toptxn = txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ if (txn->toptxn != NULL)
+ toptxn = txn->toptxn;
change->lsn = lsn;
change->txn = txn;
@@ -668,9 +671,50 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If this is a toast insert then set the corresponding bit. Otherwise, if
+ * we have toast insert bit set and this is insert/update then clear the
+ * bit.
+ */
+ if (toast_insert)
+ toptxn->txn_flags |= RBTXN_HAS_TOAST_INSERT;
+ else if (rbtxn_has_toast_insert(txn) &&
+ ((change->action == REORDER_BUFFER_CHANGE_INSERT) ||
+ (change->action == REORDER_BUFFER_CHANGE_UPDATE)))
+ {
+ toptxn->txn_flags &= ~RBTXN_HAS_TOAST_INSERT;
+ can_stream = true;
+ }
+
+ /*
+ * If this is a speculative insert then set the corresponding bit.
+ * Otherwise, if we have speculative insert bit set and this is spec confirm
+ * record then clear the bit.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT)
+ toptxn->txn_flags |= RBTXN_HAS_SPEC_INSERT;
+ else if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM)
+ {
+ toptxn->txn_flags &= ~RBTXN_HAS_SPEC_INSERT;
+ can_stream = true;
+ }
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, true);
+ /*
+ * If streaming is enable and we have serialized this transaction because
+ * it had incomplete tuple. So if now we have got the complete tuple we
+ * can stream it.
+ */
+ if (ReorderBufferCanStream(rb) && can_stream && rbtxn_is_serialized(toptxn)
+ && !rbtxn_has_toast_insert(txn) && !rbtxn_has_spec_insert(txn))
+ {
+ ReorderBufferStreamTXN(rb, toptxn);
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
+ }
+
/* check the memory limits and evict something if needed */
ReorderBufferCheckMemoryLimit(rb);
}
@@ -700,7 +744,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
change->data.msg.message = palloc(message_size);
memcpy(change->data.msg.message, message, message_size);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -1862,8 +1906,8 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* disk.
*/
dlist_delete(&change->node);
- ReorderBufferToastAppendChunk(rb, txn, relation,
- change);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
}
change_done:
@@ -2456,7 +2500,7 @@ ReorderBufferAddSnapshot(ReorderBuffer *rb, TransactionId xid,
change->data.snapshot = snap;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2505,7 +2549,7 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
change->data.command_id = cid;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2528,6 +2572,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
Size sz;
ReorderBufferTXN *txn;
+ ReorderBufferTXN *toptxn = NULL;
Assert(change->txn);
@@ -2542,8 +2587,13 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
txn = change->txn;
/* if subxact, and streaming supported, use the toplevel instead */
- if (txn->toptxn && ReorderBufferCanStream(rb))
- txn = txn->toptxn;
+ if (ReorderBufferCanStream(rb))
+ {
+ if (txn->toptxn)
+ toptxn = txn->toptxn;
+ else
+ toptxn = txn;
+ }
sz = ReorderBufferChangeSize(change);
@@ -2551,12 +2601,20 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
txn->size += sz;
rb->size += sz;
+
+ /* Update the total size in the top transaction. */
+ if (toptxn)
+ toptxn->total_size += sz;
}
else
{
Assert((rb->size >= sz) && (txn->size >= sz));
txn->size -= sz;
rb->size -= sz;
+
+ /* Update the total size in the top transaction. */
+ if (toptxn)
+ toptxn->total_size -= sz;
}
Assert(txn->size <= rb->size);
@@ -2617,7 +2675,7 @@ ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
memcpy(change->data.inval.invalidations, msgs,
sizeof(SharedInvalidationMessage) * nmsgs);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -2804,15 +2862,16 @@ ReorderBufferLargestTopTXN(ReorderBuffer *rb)
txn = dlist_container(ReorderBufferTXN, node, iter.cur);
- /* if the current transaction is larger, remember it */
- if ((!largest) || (txn->size > largest->size))
- largest = txn;
+ /*
+ * if the current transaction is larger and doesn't have incomplete data
+ * remember it.
+ */
+ if (((!largest) || (txn->total_size > largest->total_size)) &&
+ ((txn->size > 0) && !rbtxn_has_toast_insert(txn) &&
+ !rbtxn_has_spec_insert(txn)))
+ largest = txn;
}
- Assert(largest);
- Assert(largest->size > 0);
- Assert(largest->size <= rb->size);
-
return largest;
}
@@ -2830,66 +2889,51 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
ReorderBufferTXN *txn;
- /* bail out if we haven't exceeded the memory limit */
- if (rb->size < logical_decoding_work_mem * 1024L)
- return;
-
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by streaming, if supported. Otherwise spill to disk.
- */
- if (ReorderBufferCanStream(rb))
+ /* Loop until we reach under the memory limit. */
+ while (rb->size >= logical_decoding_work_mem * 1024L)
{
/*
- * Pick the largest toplevel transaction and evict it from memory by
- * streaming the already decoded part.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTopTXN(rb);
+ if (ReorderBufferCanStream(rb) &&
+ (txn = ReorderBufferLargestTopTXN(rb)) != NULL)
+ {
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->total_size > 0);
+ Assert(rb->size >= txn->total_size);
- /* we know there has to be one, because the size is not zero */
- Assert(txn && !txn->toptxn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- ReorderBufferStreamTXN(rb, txn);
- }
- else
- {
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
- */
- txn = ReorderBufferLargestTXN(rb);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- /* we know there has to be one, because the size is not zero */
- Assert(txn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
- ReorderBufferSerializeTXN(rb, txn);
+ /*
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
+ */
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
}
- /*
- * After eviction, the transaction should have no entries in memory, and
- * should use 0 bytes for changes.
- *
- * XXX Checking the size is fine for both cases - spill to disk and
- * streaming. But for streaming we should really check nentries_mem for
- * all subtransactions too.
- */
- Assert(txn->size == 0);
- Assert(txn->nentries_mem == 0);
-
- /*
- * And furthermore, evicting the transaction should get us below the
- * memory limit again - it is not possible that we're still exceeding the
- * memory limit after evicting the transaction.
- *
- * This follows from the simple fact that the selected transaction is at
- * least as large as the most recent change (which caused us to go over
- * the memory limit). So by evicting it we're definitely back below the
- * memory limit.
- */
Assert(rb->size < logical_decoding_work_mem * 1024L);
}
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 95d18cd..aa17f7d 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -67,6 +67,7 @@
#define XLH_INSERT_LAST_IN_MULTI (1<<1)
#define XLH_INSERT_IS_SPECULATIVE (1<<2)
#define XLH_INSERT_CONTAINS_NEW_TUPLE (1<<3)
+#define XLH_INSERT_ON_TOAST_RELATION (1<<4)
/*
* xl_heap_update flag values, 8 bits are available.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 603f325..ba2ab71 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -172,6 +172,8 @@ typedef struct ReorderBufferChange
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
#define RBTXN_IS_STREAMED 0x0008
+#define RBTXN_HAS_TOAST_INSERT 0x0010
+#define RBTXN_HAS_SPEC_INSERT 0x0020
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -191,6 +193,17 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/* This transaction's changes has toast insert, without main table insert. */
+#define rbtxn_has_toast_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_TOAST_INSERT) != 0 \
+
+/*
+ * This transaction's changes has speculative insert, without speculative
+ * confirm.
+ */
+#define rbtxn_has_spec_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_SPEC_INSERT) != 0 \
+
/*
* Has this transaction been streamed to downstream?
*
@@ -199,10 +212,6 @@ typedef struct ReorderBufferChange
* which case we'd have nentries==0 for the toplevel one, which would say
* nothing about the streaming. So we maintain this flag, but only for the
* toplevel transaction.)
- *
- * Note: We never do both stream and serialize a transaction (we only spill
- * to disk when streaming is not supported by the plugin), so only one of
- * those two flags may be set at any given time.
*/
#define rbtxn_is_streamed(txn) \
( \
@@ -355,6 +364,9 @@ typedef struct ReorderBufferTXN
* Size of this transaction (changes currently in memory, in bytes).
*/
Size size;
+
+ /* Size of top-transaction including sub-transactions. */
+ Size total_size;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -545,7 +557,9 @@ void ReorderBufferReturnChange(ReorderBuffer *, ReorderBufferChange *);
Oid *ReorderBufferGetRelids(ReorderBuffer *, int nrelids);
void ReorderBufferReturnRelids(ReorderBuffer *, Oid *relids);
-void ReorderBufferQueueChange(ReorderBuffer *, TransactionId, XLogRecPtr lsn, ReorderBufferChange *);
+void ReorderBufferQueueChange(ReorderBuffer *, TransactionId,
+ XLogRecPtr lsn, ReorderBufferChange *,
+ bool incomplte_data);
void ReorderBufferQueueMessage(ReorderBuffer *, TransactionId, Snapshot snapshot, XLogRecPtr lsn,
bool transactional, const char *prefix,
Size message_size, const char *message);
--
1.8.3.1
v15-0007-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From 54abdb2e87b0d65b124b59b27e918e4f64aa95a8 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 2 Apr 2020 13:19:29 +0530
Subject: [PATCH v15 07/11] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 6562cc4..d8bf587 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2063,6 +2063,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d406ea8..65d650d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -788,7 +788,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 801fdc5..845d820 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3282,6 +3286,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index eacea12..d0028d9 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1333,7 +1333,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1354,7 +1354,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2396,6 +2397,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3237,7 +3241,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3295,6 +3299,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillCount;
int64 spillBytes;
bool is_sync_standby;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
int j;
@@ -3320,6 +3327,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
/*
@@ -3422,6 +3432,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3670,11 +3685,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4bce3ad..9fb1ffe 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5237,9 +5237,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6d65986..603f325 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -517,15 +517,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 734acec..b997d17 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -83,6 +83,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index ac31840..68e2deb 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2005,9 +2005,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_slru| SELECT s.name,
s.blks_zeroed,
--
1.8.3.1
v15-0011-Provide-new-api-to-get-the-streaming-changes.patch (application/octet-stream, 4.7K) download
From d9b8f3ab04980d57a2df7f120b7c522c381935db Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Wed, 22 Apr 2020 16:33:07 +0530
Subject: [PATCH v15 11/11] Provide new api to get the streaming changes
---
src/backend/catalog/system_views.sql | 8 ++++++++
src/backend/replication/logical/logicalfuncs.c | 23 ++++++++++++++++++-----
src/include/catalog/pg_proc.dat | 9 +++++++++
3 files changed, 35 insertions(+), 5 deletions(-)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 65d650d..d9ab14b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1242,6 +1242,14 @@ LANGUAGE INTERNAL
VOLATILE ROWS 1000 COST 1000
AS 'pg_logical_slot_get_changes';
+CREATE OR REPLACE FUNCTION pg_logical_slot_get_streaming_changes(
+ IN slot_name name, IN upto_lsn pg_lsn, IN upto_nchanges int, VARIADIC options text[] DEFAULT '{}',
+ OUT lsn pg_lsn, OUT xid xid, OUT data text)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+VOLATILE ROWS 1000 COST 1000
+AS 'pg_logical_slot_get_streaming_changes';
+
CREATE OR REPLACE FUNCTION pg_logical_slot_peek_changes(
IN slot_name name, IN upto_lsn pg_lsn, IN upto_nchanges int, VARIADIC options text[] DEFAULT '{}',
OUT lsn pg_lsn, OUT xid xid, OUT data text)
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index f5384f1..7561141 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -108,7 +108,8 @@ check_permissions(void)
* Helper function for the various SQL callable logical decoding functions.
*/
static Datum
-pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool binary)
+pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm,
+ bool binary, bool streaming)
{
Name name;
XLogRecPtr upto_lsn;
@@ -237,6 +238,9 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
LogicalOutputPrepareWrite,
LogicalOutputWrite, NULL);
+ /* If called has not asked for streaming changes then disable it. */
+ ctx->streaming &= streaming;
+
MemoryContextSwitchTo(oldcontext);
/*
@@ -347,7 +351,16 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
Datum
pg_logical_slot_get_changes(PG_FUNCTION_ARGS)
{
- return pg_logical_slot_get_changes_guts(fcinfo, true, false);
+ return pg_logical_slot_get_changes_guts(fcinfo, true, false, false);
+}
+
+/*
+ * SQL function to get the streaming changes as text, consuming the data.
+ */
+Datum
+pg_logical_slot_get_streaming_changes(PG_FUNCTION_ARGS)
+{
+ return pg_logical_slot_get_changes_guts(fcinfo, true, false, true);
}
/*
@@ -356,7 +369,7 @@ pg_logical_slot_get_changes(PG_FUNCTION_ARGS)
Datum
pg_logical_slot_peek_changes(PG_FUNCTION_ARGS)
{
- return pg_logical_slot_get_changes_guts(fcinfo, false, false);
+ return pg_logical_slot_get_changes_guts(fcinfo, false, false, false);
}
/*
@@ -365,7 +378,7 @@ pg_logical_slot_peek_changes(PG_FUNCTION_ARGS)
Datum
pg_logical_slot_get_binary_changes(PG_FUNCTION_ARGS)
{
- return pg_logical_slot_get_changes_guts(fcinfo, true, true);
+ return pg_logical_slot_get_changes_guts(fcinfo, true, true, false);
}
/*
@@ -374,7 +387,7 @@ pg_logical_slot_get_binary_changes(PG_FUNCTION_ARGS)
Datum
pg_logical_slot_peek_binary_changes(PG_FUNCTION_ARGS)
{
- return pg_logical_slot_get_changes_guts(fcinfo, false, true);
+ return pg_logical_slot_get_changes_guts(fcinfo, false, true, false);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9fb1ffe..3dfc5c1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10117,6 +10117,15 @@
proargmodes => '{i,i,i,v,o,o,o}',
proargnames => '{slot_name,upto_lsn,upto_nchanges,options,lsn,xid,data}',
prosrc => 'pg_logical_slot_get_binary_changes' },
+{ oid => '6150', descr => 'get streaming changes from replication slot',
+ proname => 'pg_logical_slot_get_streaming_changes', procost => '1000',
+ prorows => '1000', provariadic => 'text', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', proparallel => 'u',
+ prorettype => 'record', proargtypes => 'name pg_lsn int4 _text',
+ proallargtypes => '{name,pg_lsn,int4,_text,pg_lsn,xid,text}',
+ proargmodes => '{i,i,i,v,o,o,o}',
+ proargnames => '{slot_name,upto_lsn,upto_nchanges,options,lsn,xid,data}',
+ prosrc => 'pg_logical_slot_get_streaming_changes' },
{ oid => '3784', descr => 'peek at changes from replication slot',
proname => 'pg_logical_slot_peek_changes', procost => '1000',
prorows => '1000', provariadic => 'text', proisstrict => 'f',
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-21 12:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 14:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-22 16:01 ` Erik Rijkers <er@xs4all.nl>
2020-04-23 03:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2020-04-22 16:01 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-04-22 16:49, Dilip Kumar wrote:
> On Tue, Apr 21, 2020 at 5:30 PM Dilip Kumar <dilipbalaut@gmail.com>
> wrote:
>>
>> >
>> > (by the way: this build's regression tests 'ddl', 'toast', and
>> > 'spill' fail)
>>
>> Yeah, this is a. known issue, actually, while streaming the
>> transaction the output message is changed. I have a plan to work on
>> this part.
>
> I have fixed this part. Basically, now, I have created a separate
> function to get the streaming changes
> 'pg_logical_slot_get_streaming_changes'. So the default function
> pg_logical_slot_get_changes will work as it is and test decoding test
> cases will not fail.
The 'ddl' one is apparently not quite fixed - I get this in (cd
contrib; make check)' (in both assert-enabled and non-assert-enabled
build)
grep -A7 -B7 make.check_contrib.out:
contrib/make.check_contrib.out-============== initializing database
system ==============
contrib/make.check_contrib.out-============== starting postmaster
==============
contrib/make.check_contrib.out-running on port 64464 with PID 9175
contrib/make.check_contrib.out-============== creating database
"contrib_regression" ==============
contrib/make.check_contrib.out-CREATE DATABASE
contrib/make.check_contrib.out-ALTER DATABASE
contrib/make.check_contrib.out-============== running regression test
queries ==============
contrib/make.check_contrib.out:test ddl ...
FAILED 840 ms
contrib/make.check_contrib.out-test xact ... ok
24 ms
contrib/make.check_contrib.out-test rewrite ... ok
187 ms
contrib/make.check_contrib.out-test toast ... ok
851 ms
contrib/make.check_contrib.out-test permissions ... ok
26 ms
contrib/make.check_contrib.out-test decoding_in_xact ... ok
31 ms
contrib/make.check_contrib.out-test decoding_into_rel ... ok
25 ms
contrib/make.check_contrib.out-test binary ... ok
12 ms
Otherwise patches apply and build OK so will go run some tests...
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-21 12:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 14:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 16:01 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2020-04-23 03:24 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-23 08:58 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-04-23 03:24 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Apr 22, 2020 at 9:31 PM Erik Rijkers <er@xs4all.nl> wrote:
>
> On 2020-04-22 16:49, Dilip Kumar wrote:
> > On Tue, Apr 21, 2020 at 5:30 PM Dilip Kumar <dilipbalaut@gmail.com>
> > wrote:
> >>
> >> >
> >> > (by the way: this build's regression tests 'ddl', 'toast', and
> >> > 'spill' fail)
> >>
> >> Yeah, this is a. known issue, actually, while streaming the
> >> transaction the output message is changed. I have a plan to work on
> >> this part.
> >
> > I have fixed this part. Basically, now, I have created a separate
> > function to get the streaming changes
> > 'pg_logical_slot_get_streaming_changes'. So the default function
> > pg_logical_slot_get_changes will work as it is and test decoding test
> > cases will not fail.
>
> The 'ddl' one is apparently not quite fixed - I get this in (cd
> contrib; make check)' (in both assert-enabled and non-assert-enabled
> build)
Can you send me the contrib/test_decoding/regression.diffs file?
> grep -A7 -B7 make.check_contrib.out:
>
> contrib/make.check_contrib.out-============== initializing database
> system ==============
> contrib/make.check_contrib.out-============== starting postmaster
> ==============
> contrib/make.check_contrib.out-running on port 64464 with PID 9175
> contrib/make.check_contrib.out-============== creating database
> "contrib_regression" ==============
> contrib/make.check_contrib.out-CREATE DATABASE
> contrib/make.check_contrib.out-ALTER DATABASE
> contrib/make.check_contrib.out-============== running regression test
> queries ==============
> contrib/make.check_contrib.out:test ddl ...
> FAILED 840 ms
> contrib/make.check_contrib.out-test xact ... ok
> 24 ms
> contrib/make.check_contrib.out-test rewrite ... ok
> 187 ms
> contrib/make.check_contrib.out-test toast ... ok
> 851 ms
> contrib/make.check_contrib.out-test permissions ... ok
> 26 ms
> contrib/make.check_contrib.out-test decoding_in_xact ... ok
> 31 ms
> contrib/make.check_contrib.out-test decoding_into_rel ... ok
> 25 ms
> contrib/make.check_contrib.out-test binary ... ok
> 12 ms
>
> Otherwise patches apply and build OK so will go run some tests...
Thanks
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-21 12:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 14:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 16:01 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-23 03:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-23 08:58 ` Erik Rijkers <er@xs4all.nl>
2020-04-24 06:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Erik Rijkers @ 2020-04-23 08:58 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On 2020-04-23 05:24, Dilip Kumar wrote:
> On Wed, Apr 22, 2020 at 9:31 PM Erik Rijkers <er@xs4all.nl> wrote:
>>
>> The 'ddl' one is apparently not quite fixed - I get this in (cd
>> contrib; make check)' (in both assert-enabled and non-assert-enabled
>> build)
>
> Can you send me the contrib/test_decoding/regression.diffs file?
Attached.
Below is the patch list, in case that was unclear
20200422/v15-0001-Immediately-WAL-log-assignments.patch
+
20200422/v15-0002-Issue-individual-invalidations-with-wal_level-lo.patch+
20200422/v15-0003-Extend-the-output-plugin-API-with-stream-methods.patch+
20200422/v15-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch+
20200422/v15-0005-Implement-streaming-mode-in-ReorderBuffer.patch
+
20200422/v15-0006-Add-support-for-streaming-to-built-in-replicatio.patch+
20200422/v15-0007-Track-statistics-for-streaming.patch
+
20200422/v15-0008-Enable-streaming-for-all-subscription-TAP-tests.patch
+
20200422/v15-0009-Add-TAP-test-for-streaming-vs.-DDL.patch
+
20200422/v15-0010-Bugfix-handling-of-incomplete-toast-tuple.patch
+
20200422/v15-0011-Provide-new-api-to-get-the-streaming-changes.patch
+
20200414/bugfix_in_schema_sent.patch
>> grep -A7 -B7 make.check_contrib.out:
>>
>> contrib/make.check_contrib.out-============== initializing database
>> system ==============
>> contrib/make.check_contrib.out-============== starting postmaster
>> ==============
>> contrib/make.check_contrib.out-running on port 64464 with PID 9175
>> contrib/make.check_contrib.out-============== creating database
>> "contrib_regression" ==============
>> contrib/make.check_contrib.out-CREATE DATABASE
>> contrib/make.check_contrib.out-ALTER DATABASE
>> contrib/make.check_contrib.out-============== running regression test
>> queries ==============
>> contrib/make.check_contrib.out:test ddl ...
>> FAILED 840 ms
>> contrib/make.check_contrib.out-test xact ...
>> ok
>> 24 ms
>> contrib/make.check_contrib.out-test rewrite ...
>> ok
>> 187 ms
>> contrib/make.check_contrib.out-test toast ...
>> ok
>> 851 ms
>> contrib/make.check_contrib.out-test permissions ...
>> ok
>> 26 ms
>> contrib/make.check_contrib.out-test decoding_in_xact ...
>> ok
>> 31 ms
>> contrib/make.check_contrib.out-test decoding_into_rel ...
>> ok
>> 25 ms
>> contrib/make.check_contrib.out-test binary ...
>> ok
>> 12 ms
>>
>> Otherwise patches apply and build OK so will go run some tests...
>
> Thanks
>
>
> --
> Regards,
> Dilip Kumar
> EnterpriseDB: http://www.enterprisedb.com
Attachments:
regression.diffs (text/x-diff, 347.3K) download
diff -U3 /home/aardvark/pg_stuff/pg_sandbox/pgsql.large_logical/contrib/test_decoding/expected/ddl.out /home/aardvark/pg_stuff/pg_sandbox/pgsql.large_logical/contrib/test_decoding/results/ddl.out
--- /home/aardvark/pg_stuff/pg_sandbox/pgsql.large_logical/contrib/test_decoding/expected/ddl.out 2020-04-22 18:08:28.166822219 +0200
+++ /home/aardvark/pg_stuff/pg_sandbox/pgsql.large_logical/contrib/test_decoding/results/ddl.out 2020-04-22 18:18:55.996887367 +0200
@@ -245,15 +245,7 @@
FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
GROUP BY substring(data, 1, 24)
ORDER BY 1,2;
- count | min | max
--------+-----------------------------------------------------------------------+------------------------------------------------------------------------
- 1 | BEGIN | BEGIN
- 1 | COMMIT | COMMIT
- 1 | message: transactional: 1 prefix: test, sz: 14 content:tx logical msg | message: transactional: 1 prefix: test, sz: 14 content:tx logical msg
- 1 | table public.tr_oddlength: INSERT: id[text]:'ab' data[text]:'foo' | table public.tr_oddlength: INSERT: id[text]:'ab' data[text]:'foo'
- 20467 | table public.tr_etoomuch: DELETE: id[integer]:1 | table public.tr_etoomuch: UPDATE: id[integer]:9999 data[integer]:-9999
-(5 rows)
-
+ERROR: invalid memory alloc request size 94119198201896
-- check updates of primary keys work correctly
BEGIN;
CREATE TABLE spoolme AS SELECT g.i FROM generate_series(1, 5000) g(i);
@@ -266,13 +258,7 @@
SELECT data
FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1')
WHERE data ~ 'UPDATE';
- data
--------------------------------------------------------------------------------------------------------------
- table public.tr_etoomuch: UPDATE: old-key: id[integer]:5000 new-tuple: id[integer]:-5000 data[integer]:5000
- table public.tr_oddlength: UPDATE: old-key: id[text]:'ab' new-tuple: id[text]:'x' data[text]:'quux'
- table public.tr_oddlength: UPDATE: old-key: id[text]:'x' new-tuple: id[text]:'yy' data[text]:'a'
-(3 rows)
-
+ERROR: invalid memory alloc request size 94119197988536
-- check that a large, spooled, upsert works
INSERT INTO tr_etoomuch (id, data)
SELECT g.i, -g.i FROM generate_series(8000, 12000) g(i)
@@ -281,14 +267,7 @@
FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1') WITH ORDINALITY
GROUP BY 1
ORDER BY min(ordinality);
- substring | count
--------------------------------+-------
- BEGIN | 1
- table public.tr_etoomuch: UPD | 2235
- table public.tr_etoomuch: INS | 1766
- COMMIT | 1
-(4 rows)
-
+ERROR: invalid memory alloc request size 94119198201896
/*
* check whether we decode subtransactions correctly in relation with each
* other
@@ -310,18 +289,7 @@
RELEASE SAVEPOINT b;
COMMIT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-----------------------------------------------------------------------
- BEGIN
- table public.tr_sub: INSERT: id[integer]:1 path[text]:'1-top-#1'
- table public.tr_sub: INSERT: id[integer]:2 path[text]:'1-top-1-#1'
- table public.tr_sub: INSERT: id[integer]:3 path[text]:'1-top-1-#2'
- table public.tr_sub: INSERT: id[integer]:4 path[text]:'1-top-2-1-#1'
- table public.tr_sub: INSERT: id[integer]:5 path[text]:'1-top-2-1-#2'
- table public.tr_sub: INSERT: id[integer]:6 path[text]:'1-top-2-#1'
- COMMIT
-(8 rows)
-
+ERROR: invalid memory alloc request size 94119197980328
-- check that we handle xlog assignments correctly
BEGIN;
-- nest 80 subtxns
@@ -349,16 +317,7 @@
INSERT INTO tr_sub(path) VALUES ('2-top-#1');
COMMIT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-------------------------------------------------------------------------
- BEGIN
- table public.tr_sub: INSERT: id[integer]:7 path[text]:'2-top-1...--#1'
- table public.tr_sub: INSERT: id[integer]:8 path[text]:'2-top-1...--#2'
- table public.tr_sub: INSERT: id[integer]:9 path[text]:'2-top-1...--#3'
- table public.tr_sub: INSERT: id[integer]:10 path[text]:'2-top-#1'
- COMMIT
-(6 rows)
-
+ERROR: invalid memory alloc request size 94119197980328
-- make sure rollbacked subtransactions aren't decoded
BEGIN;
INSERT INTO tr_sub(path) VALUES ('3-top-2-#1');
@@ -370,15 +329,7 @@
INSERT INTO tr_sub(path) VALUES ('3-top-2-#2');
COMMIT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
------------------------------------------------------------------------
- BEGIN
- table public.tr_sub: INSERT: id[integer]:11 path[text]:'3-top-2-#1'
- table public.tr_sub: INSERT: id[integer]:12 path[text]:'3-top-2-1-#1'
- table public.tr_sub: INSERT: id[integer]:14 path[text]:'3-top-2-#2'
- COMMIT
-(5 rows)
-
+ERROR: invalid memory alloc request size 94119197980328
-- test whether a known, but not yet logged toplevel xact, followed by a
-- subxact commit is handled correctly
BEGIN;
@@ -399,16 +350,7 @@
INSERT INTO tr_sub(path) VALUES ('5-top-1-#1');
COMMIT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
----------------------------------------------------------------------
- BEGIN
- table public.tr_sub: INSERT: id[integer]:15 path[text]:'4-top-1-#1'
- COMMIT
- BEGIN
- table public.tr_sub: INSERT: id[integer]:16 path[text]:'5-top-1-#1'
- COMMIT
-(6 rows)
-
+ERROR: invalid memory alloc request size 94119197980328
-- check that DDL in aborted subtransactions handled correctly
CREATE TABLE tr_sub_ddl(data int);
BEGIN;
@@ -420,13 +362,7 @@
INSERT INTO tr_sub_ddl VALUES(43);
COMMIT;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
---------------------------------------------------
- BEGIN
- table public.tr_sub_ddl: INSERT: data[bigint]:43
- COMMIT
-(3 rows)
-
+ERROR: invalid memory alloc request size 94119198078792
/*
* Check whether treating a table as a catalog table works somewhat
*/
@@ -497,22 +433,7 @@
INSERT INTO replication_metadata(relation, options)
VALUES ('zaphod', NULL);
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-------------------------------------------------------------------------------------------------------------------------------------
- BEGIN
- table public.replication_metadata: INSERT: id[integer]:1 relation[name]:'foo' options[text[]]:'{a,b}'
- COMMIT
- BEGIN
- table public.replication_metadata: INSERT: id[integer]:2 relation[name]:'bar' options[text[]]:'{a,b}'
- COMMIT
- BEGIN
- table public.replication_metadata: INSERT: id[integer]:3 relation[name]:'blub' options[text[]]:null
- COMMIT
- BEGIN
- table public.replication_metadata: INSERT: id[integer]:4 relation[name]:'zaphod' options[text[]]:null rewritemeornot[integer]:null
- COMMIT
-(12 rows)
-
+ERROR: invalid memory alloc request size 94119206549464
/*
* check whether we handle updates/deletes correct with & without a pkey
*/
@@ -585,113 +506,7 @@
SET toasted_col1 = (SELECT string_agg(g.i::text, '') FROM generate_series(1, 2000) g(i))
WHERE id = 1;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- BEGIN
- table public.table_without_key: INSERT: id[integer]:1 data[integer]:1
- table public.table_without_key: INSERT: id[integer]:2 data[integer]:2
- COMMIT
- BEGIN
- table public.table_without_key: DELETE: (no-tuple-data)
- COMMIT
- BEGIN
- table public.table_without_key: UPDATE: id[integer]:2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_without_key: UPDATE: id[integer]:-2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_without_key: UPDATE: id[integer]:2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_without_key: UPDATE: old-key: id[integer]:2 data[integer]:3 new-tuple: id[integer]:-2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_without_key: UPDATE: old-key: id[integer]:-2 data[integer]:3 new-tuple: id[integer]:2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_without_key: UPDATE: old-key: id[integer]:2 data[integer]:3 new-tuple: id[integer]:-2 data[integer]:3 new_column[text]:null
- COMMIT
- BEGIN
- table public.table_without_key: UPDATE: old-key: id[integer]:-2 data[integer]:3 new-tuple: id[integer]:2 data[integer]:3 new_column[text]:'someval'
- COMMIT
- BEGIN
- table public.table_without_key: DELETE: id[integer]:2 data[integer]:3 new_column[text]:'someval'
- COMMIT
- BEGIN
- table public.table_with_pkey: INSERT: id[integer]:1 data[integer]:1
- table public.table_with_pkey: INSERT: id[integer]:2 data[integer]:2
- COMMIT
- BEGIN
- table public.table_with_pkey: DELETE: id[integer]:1
- COMMIT
- BEGIN
- table public.table_with_pkey: UPDATE: id[integer]:2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_pkey: UPDATE: old-key: id[integer]:2 new-tuple: id[integer]:-2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_pkey: UPDATE: old-key: id[integer]:-2 new-tuple: id[integer]:2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_pkey: UPDATE: old-key: id[integer]:2 new-tuple: id[integer]:-2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_pkey: UPDATE: old-key: id[integer]:-2 new-tuple: id[integer]:2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_pkey: DELETE: id[integer]:2
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: INSERT: id[integer]:1 data[integer]:1
- table public.table_with_unique_not_null: INSERT: id[integer]:2 data[integer]:2
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: DELETE: (no-tuple-data)
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: UPDATE: id[integer]:2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: UPDATE: id[integer]:-2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: UPDATE: id[integer]:2 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: DELETE: (no-tuple-data)
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: INSERT: id[integer]:3 data[integer]:1
- table public.table_with_unique_not_null: INSERT: id[integer]:4 data[integer]:2
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: DELETE: id[integer]:3
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: UPDATE: id[integer]:4 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: UPDATE: old-key: id[integer]:4 new-tuple: id[integer]:-4 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: UPDATE: old-key: id[integer]:-4 new-tuple: id[integer]:4 data[integer]:3
- COMMIT
- BEGIN
- table public.table_with_unique_not_null: DELETE: id[integer]:4
- COMMIT
- BEGIN
- table public.toasttable: INSERT: id[integer]:1 toasted_col1[text]:'12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793179417951796179717981799180018011802180318041805180618071808180918101811181218131814181518161817181818191820182118221823182418251826182718281829183018311832183318341835183618371838183918401841184218431844184518461847184818491850185118521853185418551856185718581859186018611862186318641865186618671868186918701871187218731874187518761877187818791880188118821883188418851886188718881889189018911892189318941895189618971898189919001901190219031904190519061907190819091910191119121913191419151916191719181919192019211922192319241925192619271928192919301931193219331934193519361937193819391940194119421943194419451946194719481949195019511952195319541955195619571958195919601961196219631964196519661967196819691970197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000' rand1[double precision]:79 toasted_col2[text]:null rand2[double precision]:1578
- COMMIT
- BEGIN
- table public.toasttable: INSERT: id[integer]:2 toasted_col1[text]:null rand1[double precision]:3077 toasted_col2[text]:'0001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500000100020003000400050006000700080009001000110012001300140015001600170018001900200021002200230024002500260027002800290030003100320033003400350036003700380039004000410042004300440045004600470048004900500051005200530054005500560057005800590060006100620063006400650066006700680069007000710072007300740075007600770078007900800081008200830084008500860087008800890090009100920093009400950096009700980099010001010102010301040105010601070108010901100111011201130114011501160117011801190120012101220123012401250126012701280129013001310132013301340135013601370138013901400141014201430144014501460147014801490150015101520153015401550156015701580159016001610162016301640165016601670168016901700171017201730174017501760177017801790180018101820183018401850186018701880189019001910192019301940195019601970198019902000201020202030204020502060207020802090210021102120213021402150216021702180219022002210222022302240225022602270228022902300231023202330234023502360237023802390240024102420243024402450246024702480249025002510252025302540255025602570258025902600261026202630264026502660267026802690270027102720273027402750276027702780279028002810282028302840285028602870288028902900291029202930294029502960297029802990300030103020303030403050306030703080309031003110312031303140315031603170318031903200321032203230324032503260327032803290330033103320333033403350336033703380339034003410342034303440345034603470348034903500351035203530354035503560357035803590360036103620363036403650366036703680369037003710372037303740375037603770378037903800381038203830384038503860387038803890390039103920393039403950396039703980399040004010402040304040405040604070408040904100411041204130414041504160417041804190420042104220423042404250426042704280429043004310432043304340435043604370438043904400441044204430444044504460447044804490450045104520453045404550456045704580459046004610462046304640465046604670468046904700471047204730474047504760477047804790480048104820483048404850486048704880489049004910492049304940495049604970498049905000001000200030004000500060007000800090010001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500360037003800390040004100420043004400450046004700480049005000510052005300540055005600570058005900600061006200630064006500660067006800690070007100720073007400750076007700780079008000810082008300840085008600870088008900900091009200930094009500960097009800990100010101020103010401050106010701080109011001110112011301140115011601170118011901200121012201230124012501260127012801290130013101320133013401350136013701380139014001410142014301440145014601470148014901500151015201530154015501560157015801590160016101620163016401650166016701680169017001710172017301740175017601770178017901800181018201830184018501860187018801890190019101920193019401950196019701980199020002010202020302040205020602070208020902100211021202130214021502160217021802190220022102220223022402250226022702280229023002310232023302340235023602370238023902400241024202430244024502460247024802490250025102520253025402550256025702580259026002610262026302640265026602670268026902700271027202730274027502760277027802790280028102820283028402850286028702880289029002910292029302940295029602970298029903000301030203030304030503060307030803090310031103120313031403150316031703180319032003210322032303240325032603270328032903300331033203330334033503360337033803390340034103420343034403450346034703480349035003510352035303540355035603570358035903600361036203630364036503660367036803690370037103720373037403750376037703780379038003810382038303840385038603870388038903900391039203930394039503960397039803990400040104020403040404050406040704080409041004110412041304140415041604170418041904200421042204230424042504260427042804290430043104320433043404350436043704380439044004410442044304440445044604470448044904500451045204530454045504560457045804590460046104620463046404650466046704680469047004710472047304740475047604770478047904800481048204830484048504860487048804890490049104920493049404950496049704980499050000010002000300040005000600070008000900100011001200130014001500160017001800190020002100220023002400250026002700280029003000310032003300340035003600370038003900400041004200430044004500460047004800490050005100520053005400550056005700580059006000610062006300640065006600670068006900700071007200730074007500760077007800790080008100820083008400850086008700880089009000910092009300940095009600970098009901000101010201030104010501060107010801090110011101120113011401150116011701180119012001210122012301240125012601270128012901300131013201330134013501360137013801390140014101420143014401450146014701480149015001510152015301540155015601570158015901600161016201630164016501660167016801690170017101720173017401750176017701780179018001810182018301840185018601870188018901900191019201930194019501960197019801990200020102020203020402050206020702080209021002110212021302140215021602170218021902200221022202230224022502260227022802290230023102320233023402350236023702380239024002410242024302440245024602470248024902500251025202530254025502560257025802590260026102620263026402650266026702680269027002710272027302740275027602770278027902800281028202830284028502860287028802890290029102920293029402950296029702980299030003010302030303040305030603070308030903100311031203130314031503160317031803190320032103220323032403250326032703280329033003310332033303340335033603370338033903400341034203430344034503460347034803490350035103520353035403550356035703580359036003610362036303640365036603670368036903700371037203730374037503760377037803790380038103820383038403850386038703880389039003910392039303940395039603970398039904000401040204030404040504060407040804090410041104120413041404150416041704180419042004210422042304240425042604270428042904300431043204330434043504360437043804390440044104420443044404450446044704480449045004510452045304540455045604570458045904600461046204630464046504660467046804690470047104720473047404750476047704780479048004810482048304840485048604870488048904900491049204930494049504960497049804990500' rand2[double precision]:4576
- COMMIT
- BEGIN
- table public.toasttable: UPDATE: id[integer]:1 toasted_col1[text]:'12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793179417951796179717981799180018011802180318041805180618071808180918101811181218131814181518161817181818191820182118221823182418251826182718281829183018311832183318341835183618371838183918401841184218431844184518461847184818491850185118521853185418551856185718581859186018611862186318641865186618671868186918701871187218731874187518761877187818791880188118821883188418851886188718881889189018911892189318941895189618971898189919001901190219031904190519061907190819091910191119121913191419151916191719181919192019211922192319241925192619271928192919301931193219331934193519361937193819391940194119421943194419451946194719481949195019511952195319541955195619571958195919601961196219631964196519661967196819691970197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000' rand1[double precision]:79 toasted_col2[text]:null rand2[double precision]:1578
- COMMIT
-(103 rows)
-
+ERROR: invalid memory alloc request size 94119197734824
INSERT INTO toasttable(toasted_col1) SELECT string_agg(g.i::text, '') FROM generate_series(1, 2000) g(i);
-- update of second column, first column unchanged
UPDATE toasttable
@@ -700,22 +515,10 @@
-- make sure we decode correctly even if the toast table is gone
DROP TABLE toasttable;
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- BEGIN
- table public.toasttable: INSERT: id[integer]:3 toasted_col1[text]:'12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793179417951796179717981799180018011802180318041805180618071808180918101811181218131814181518161817181818191820182118221823182418251826182718281829183018311832183318341835183618371838183918401841184218431844184518461847184818491850185118521853185418551856185718581859186018611862186318641865186618671868186918701871187218731874187518761877187818791880188118821883188418851886188718881889189018911892189318941895189618971898189919001901190219031904190519061907190819091910191119121913191419151916191719181919192019211922192319241925192619271928192919301931193219331934193519361937193819391940194119421943194419451946194719481949195019511952195319541955195619571958195919601961196219631964196519661967196819691970197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000' rand1[double precision]:6075 toasted_col2[text]:null rand2[double precision]:7574
- COMMIT
- BEGIN
- table public.toasttable: UPDATE: id[integer]:1 toasted_col1[text]:unchanged-toast-datum rand1[double precision]:79 toasted_col2[text]:'12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793179417951796179717981799180018011802180318041805180618071808180918101811181218131814181518161817181818191820182118221823182418251826182718281829183018311832183318341835183618371838183918401841184218431844184518461847184818491850185118521853185418551856185718581859186018611862186318641865186618671868186918701871187218731874187518761877187818791880188118821883188418851886188718881889189018911892189318941895189618971898189919001901190219031904190519061907190819091910191119121913191419151916191719181919192019211922192319241925192619271928192919301931193219331934193519361937193819391940194119421943194419451946194719481949195019511952195319541955195619571958195919601961196219631964196519661967196819691970197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000' rand2[double precision]:1578
- COMMIT
-(6 rows)
-
+ERROR: invalid memory alloc request size 94119197734824
-- done, free logical replication slot
SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1');
- data
-------
-(0 rows)
-
+ERROR: invalid memory alloc request size 94119197734824
SELECT pg_drop_replication_slot('regression_slot');
pg_drop_replication_slot
--------------------------
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-21 12:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 14:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 16:01 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-23 03:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-23 08:58 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
@ 2020-04-24 06:24 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Mahendra Singh Thalor <mahi6run@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-04-24 06:24 UTC (permalink / raw)
To: Erik Rijkers <er@xs4all.nl>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Thu, Apr 23, 2020 at 2:28 PM Erik Rijkers <er@xs4all.nl> wrote:
>
> On 2020-04-23 05:24, Dilip Kumar wrote:
> > On Wed, Apr 22, 2020 at 9:31 PM Erik Rijkers <er@xs4all.nl> wrote:
> >>
> >> The 'ddl' one is apparently not quite fixed - I get this in (cd
> >> contrib; make check)' (in both assert-enabled and non-assert-enabled
> >> build)
> >
> > Can you send me the contrib/test_decoding/regression.diffs file?
>
> Attached.
So from regression.diff, it appears that in failing in memory
allocation (+ERROR: invalid memory alloc request size
94119198201896). My colleague tried to reproduce this in a different
environment but there is no success so far. One more thing surprises
me is that after
(v15-0011-Provide-new-api-to-get-the-streaming-changes.patch)
actually, it should never go for the streaming path. However, we can
not ignore the fact that some of the changes might impact the
non-streaming path as well. Is it possible for you to somehow stop or
break the code and send the stack trace? One idea is by seeing the
log we can see from where the error is raised i.e MemoryContextAlloc
or palloc or some other similar function. Once we know that we can
convert that error to an assert and find the call stack.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-21 12:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 14:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 16:01 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-23 03:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-23 08:58 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-24 06:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-29 05:45 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-04-29 07:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Mahendra Singh Thalor <mahi6run@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Mahendra Singh Thalor @ 2020-04-29 05:45 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Erik Rijkers <er@xs4all.nl>; Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, 24 Apr 2020 at 11:55, Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Thu, Apr 23, 2020 at 2:28 PM Erik Rijkers <er@xs4all.nl> wrote:
> >
> > On 2020-04-23 05:24, Dilip Kumar wrote:
> > > On Wed, Apr 22, 2020 at 9:31 PM Erik Rijkers <er@xs4all.nl> wrote:
> > >>
> > >> The 'ddl' one is apparently not quite fixed - I get this in (cd
> > >> contrib; make check)' (in both assert-enabled and non-assert-enabled
> > >> build)
> > >
> > > Can you send me the contrib/test_decoding/regression.diffs file?
> >
> > Attached.
>
> So from regression.diff, it appears that in failing in memory
> allocation (+ERROR: invalid memory alloc request size
> 94119198201896). My colleague tried to reproduce this in a different
> environment but there is no success so far. One more thing surprises
> me is that after
> (v15-0011-Provide-new-api-to-get-the-streaming-changes.patch)
> actually, it should never go for the streaming path. However, we can
> not ignore the fact that some of the changes might impact the
> non-streaming path as well. Is it possible for you to somehow stop or
> break the code and send the stack trace? One idea is by seeing the
> log we can see from where the error is raised i.e MemoryContextAlloc
> or palloc or some other similar function. Once we know that we can
> convert that error to an assert and find the call stack.
>
> --
Thanks Erik for reporting this issue.
I am able to reproduce this issue(+ERROR: invalid memory alloc
request size) on the top of v16 patch set. I applied all patches(12
patches) of v16 series and then I fired "make check -i" from
"contrib/test_decoding" folder. Below is stack trace of error:
#0 0x0000560b1350902d in MemoryContextAlloc (context=0x560b14188d70,
size=94605581787992) at mcxt.c:806
#1 0x0000560b130f0ad5 in ReorderBufferRestoreChange
(rb=0x560b14188e90, txn=0x560b141baf08, data=0x560b1418a5e8 "K") at
reorderbuffer.c:3680
#2 0x0000560b130f0662 in ReorderBufferRestoreChanges
(rb=0x560b14188e90, txn=0x560b141baf08, file=0x560b1418ad10,
segno=0x560b1418ad20) at reorderbuffer.c:3564
#3 0x0000560b130e918a in ReorderBufferIterTXNInit (rb=0x560b14188e90,
txn=0x560b141baf08, iter_state=0x7ffef18b1600) at reorderbuffer.c:1186
#4 0x0000560b130eaee1 in ReorderBufferProcessTXN (rb=0x560b14188e90,
txn=0x560b141baf08, commit_lsn=25986584, snapshot_now=0x560b141b74d8,
command_id=0, streaming=false)
at reorderbuffer.c:1785
#5 0x0000560b130ecae1 in ReorderBufferCommit (rb=0x560b14188e90,
xid=508, commit_lsn=25986584, end_lsn=25989088,
commit_time=641449268431600, origin_id=0, origin_lsn=0)
at reorderbuffer.c:2315
#6 0x0000560b130d14a1 in DecodeCommit (ctx=0x560b1416ea80,
buf=0x7ffef18b19b0, parsed=0x7ffef18b1850, xid=508) at decode.c:654
#7 0x0000560b130cff98 in DecodeXactOp (ctx=0x560b1416ea80,
buf=0x7ffef18b19b0) at decode.c:261
#8 0x0000560b130cf99a in LogicalDecodingProcessRecord
(ctx=0x560b1416ea80, record=0x560b1416ee00) at decode.c:130
#9 0x0000560b130dbbbc in pg_logical_slot_get_changes_guts
(fcinfo=0x560b1417ee50, confirm=true, binary=false, streaming=false)
at logicalfuncs.c:285
#10 0x0000560b130dbe71 in pg_logical_slot_get_changes
(fcinfo=0x560b1417ee50) at logicalfuncs.c:354
#11 0x0000560b12e294d4 in ExecMakeTableFunctionResult
(setexpr=0x560b14177838, econtext=0x560b14177748,
argContext=0x560b1417ed30, expectedDesc=0x560b141814a0,
randomAccess=false) at execSRF.c:234
#12 0x0000560b12e5490f in FunctionNext (node=0x560b14177630) at
nodeFunctionscan.c:94
#13 0x0000560b12e2c108 in ExecScanFetch (node=0x560b14177630,
accessMtd=0x560b12e54836 <FunctionNext>, recheckMtd=0x560b12e54e15
<FunctionRecheck>) at execScan.c:133
#14 0x0000560b12e2c227 in ExecScan (node=0x560b14177630,
accessMtd=0x560b12e54836 <FunctionNext>, recheckMtd=0x560b12e54e15
<FunctionRecheck>) at execScan.c:199
#15 0x0000560b12e54e9b in ExecFunctionScan (pstate=0x560b14177630) at
nodeFunctionscan.c:270
#16 0x0000560b12e24e23 in ExecProcNodeFirst (node=0x560b14177630) at
execProcnode.c:450
#17 0x0000560b12e3e172 in ExecProcNode (node=0x560b14177630) at
../../../src/include/executor/executor.h:245
#18 0x0000560b12e3e998 in fetch_input_tuple (aggstate=0x560b14176f40)
at nodeAgg.c:566
#19 0x0000560b12e4398f in agg_fill_hash_table
(aggstate=0x560b14176f40) at nodeAgg.c:2518
#20 0x0000560b12e42c9a in ExecAgg (pstate=0x560b14176f40) at nodeAgg.c:2139
#21 0x0000560b12e24e23 in ExecProcNodeFirst (node=0x560b14176f40) at
execProcnode.c:450
#22 0x0000560b12e8bb58 in ExecProcNode (node=0x560b14176f40) at
../../../src/include/executor/executor.h:245
#23 0x0000560b12e8bd59 in ExecSort (pstate=0x560b14176d28) at nodeSort.c:108
#24 0x0000560b12e24e23 in ExecProcNodeFirst (node=0x560b14176d28) at
execProcnode.c:450
#25 0x0000560b12e10e71 in ExecProcNode (node=0x560b14176d28) at
../../../src/include/executor/executor.h:245
#26 0x0000560b12e15c4c in ExecutePlan (estate=0x560b14176af0,
planstate=0x560b14176d28, use_parallel_mode=false,
operation=CMD_SELECT, sendTuples=true, numberTuples=0,
direction=ForwardScanDirection, dest=0x560b1419d188,
execute_once=true) at execMain.c:1646
#27 0x0000560b12e11a19 in standard_ExecutorRun
(queryDesc=0x560b1412db10, direction=ForwardScanDirection, count=0,
execute_once=true) at execMain.c:364
#28 0x0000560b12e116e1 in ExecutorRun (queryDesc=0x560b1412db10,
direction=ForwardScanDirection, count=0, execute_once=true) at
execMain.c:308
#29 0x0000560b131f2177 in PortalRunSelect (portal=0x560b140db860,
forward=true, count=0, dest=0x560b1419d188) at pquery.c:912
#30 0x0000560b131f1b14 in PortalRun (portal=0x560b140db860,
count=9223372036854775807, isTopLevel=true, run_once=true,
dest=0x560b1419d188, altdest=0x560b1419d188,
qc=0x7ffef18b2350) at pquery.c:756
#31 0x0000560b131e550b in exec_simple_query (
query_string=0x560b14076720 "/ display results, but hide most of the
output /\nSELECT count(*), min(data), max(data)\nFROM
pg_logical_slot_get_changes('regression_slot', NULL, NULL,
'include-xids', '0', 'skip-empty-xacts', '1')\nG"...) at
postgres.c:1239
#32 0x0000560b131ee343 in PostgresMain (argc=1, argv=0x560b1409faa0,
dbname=0x560b1409f858 "contrib_regression", username=0x560b1409f830
"mahendrathalor") at postgres.c:4315
#33 0x0000560b130a325b in BackendRun (port=0x560b14096880) at postmaster.c:4510
#34 0x0000560b130a22c3 in BackendStartup (port=0x560b14096880) at
postmaster.c:4202
#35 0x0000560b1309a5cc in ServerLoop () at postmaster.c:1727
#36 0x0000560b130997c9 in PostmasterMain (argc=8, argv=0x560b1406f010)
at postmaster.c:1400
#37 0x0000560b12ee9530 in main (argc=8, argv=0x560b1406f010) at main.c:210
I have Ubuntu setup. I think, this is reproducing into Ubuntu only. I
am looking into this issue with Dilip.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
Attachments:
regression.diffs (application/octet-stream, 347.3K) download
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-21 12:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 14:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 16:01 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-23 03:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-23 08:58 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-24 06:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Mahendra Singh Thalor <mahi6run@gmail.com>
@ 2020-04-29 07:07 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-04-29 07:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Mahendra Singh Thalor @ 2020-04-29 07:07 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Erik Rijkers <er@xs4all.nl>; Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, 29 Apr 2020 at 11:15, Mahendra Singh Thalor <mahi6run@gmail.com> wrote:
>
> On Fri, 24 Apr 2020 at 11:55, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Thu, Apr 23, 2020 at 2:28 PM Erik Rijkers <er@xs4all.nl> wrote:
> > >
> > > On 2020-04-23 05:24, Dilip Kumar wrote:
> > > > On Wed, Apr 22, 2020 at 9:31 PM Erik Rijkers <er@xs4all.nl> wrote:
> > > >>
> > > >> The 'ddl' one is apparently not quite fixed - I get this in (cd
> > > >> contrib; make check)' (in both assert-enabled and non-assert-enabled
> > > >> build)
> > > >
> > > > Can you send me the contrib/test_decoding/regression.diffs file?
> > >
> > > Attached.
> >
> > So from regression.diff, it appears that in failing in memory
> > allocation (+ERROR: invalid memory alloc request size
> > 94119198201896). My colleague tried to reproduce this in a different
> > environment but there is no success so far. One more thing surprises
> > me is that after
> > (v15-0011-Provide-new-api-to-get-the-streaming-changes.patch)
> > actually, it should never go for the streaming path. However, we can
> > not ignore the fact that some of the changes might impact the
> > non-streaming path as well. Is it possible for you to somehow stop or
> > break the code and send the stack trace? One idea is by seeing the
> > log we can see from where the error is raised i.e MemoryContextAlloc
> > or palloc or some other similar function. Once we know that we can
> > convert that error to an assert and find the call stack.
> >
> > --
>
> Thanks Erik for reporting this issue.
>
> I am able to reproduce this issue(+ERROR: invalid memory alloc
> request size) on the top of v16 patch set. I applied all patches(12
> patches) of v16 series and then I fired "make check -i" from
> "contrib/test_decoding" folder. Below is stack trace of error:
>
> #0 0x0000560b1350902d in MemoryContextAlloc (context=0x560b14188d70,
> size=94605581787992) at mcxt.c:806
> #1 0x0000560b130f0ad5 in ReorderBufferRestoreChange
> (rb=0x560b14188e90, txn=0x560b141baf08, data=0x560b1418a5e8 "K") at
> reorderbuffer.c:3680
> #2 0x0000560b130f0662 in ReorderBufferRestoreChanges
> (rb=0x560b14188e90, txn=0x560b141baf08, file=0x560b1418ad10,
> segno=0x560b1418ad20) at reorderbuffer.c:3564
> #3 0x0000560b130e918a in ReorderBufferIterTXNInit (rb=0x560b14188e90,
> txn=0x560b141baf08, iter_state=0x7ffef18b1600) at reorderbuffer.c:1186
> #4 0x0000560b130eaee1 in ReorderBufferProcessTXN (rb=0x560b14188e90,
> txn=0x560b141baf08, commit_lsn=25986584, snapshot_now=0x560b141b74d8,
> command_id=0, streaming=false)
> at reorderbuffer.c:1785
> #5 0x0000560b130ecae1 in ReorderBufferCommit (rb=0x560b14188e90,
> xid=508, commit_lsn=25986584, end_lsn=25989088,
> commit_time=641449268431600, origin_id=0, origin_lsn=0)
> at reorderbuffer.c:2315
> #6 0x0000560b130d14a1 in DecodeCommit (ctx=0x560b1416ea80,
> buf=0x7ffef18b19b0, parsed=0x7ffef18b1850, xid=508) at decode.c:654
> #7 0x0000560b130cff98 in DecodeXactOp (ctx=0x560b1416ea80,
> buf=0x7ffef18b19b0) at decode.c:261
> #8 0x0000560b130cf99a in LogicalDecodingProcessRecord
> (ctx=0x560b1416ea80, record=0x560b1416ee00) at decode.c:130
> #9 0x0000560b130dbbbc in pg_logical_slot_get_changes_guts
> (fcinfo=0x560b1417ee50, confirm=true, binary=false, streaming=false)
> at logicalfuncs.c:285
> #10 0x0000560b130dbe71 in pg_logical_slot_get_changes
> (fcinfo=0x560b1417ee50) at logicalfuncs.c:354
> #11 0x0000560b12e294d4 in ExecMakeTableFunctionResult
> (setexpr=0x560b14177838, econtext=0x560b14177748,
> argContext=0x560b1417ed30, expectedDesc=0x560b141814a0,
> randomAccess=false) at execSRF.c:234
> #12 0x0000560b12e5490f in FunctionNext (node=0x560b14177630) at
> nodeFunctionscan.c:94
> #13 0x0000560b12e2c108 in ExecScanFetch (node=0x560b14177630,
> accessMtd=0x560b12e54836 <FunctionNext>, recheckMtd=0x560b12e54e15
> <FunctionRecheck>) at execScan.c:133
> #14 0x0000560b12e2c227 in ExecScan (node=0x560b14177630,
> accessMtd=0x560b12e54836 <FunctionNext>, recheckMtd=0x560b12e54e15
> <FunctionRecheck>) at execScan.c:199
> #15 0x0000560b12e54e9b in ExecFunctionScan (pstate=0x560b14177630) at
> nodeFunctionscan.c:270
> #16 0x0000560b12e24e23 in ExecProcNodeFirst (node=0x560b14177630) at
> execProcnode.c:450
> #17 0x0000560b12e3e172 in ExecProcNode (node=0x560b14177630) at
> ../../../src/include/executor/executor.h:245
> #18 0x0000560b12e3e998 in fetch_input_tuple (aggstate=0x560b14176f40)
> at nodeAgg.c:566
> #19 0x0000560b12e4398f in agg_fill_hash_table
> (aggstate=0x560b14176f40) at nodeAgg.c:2518
> #20 0x0000560b12e42c9a in ExecAgg (pstate=0x560b14176f40) at nodeAgg.c:2139
> #21 0x0000560b12e24e23 in ExecProcNodeFirst (node=0x560b14176f40) at
> execProcnode.c:450
> #22 0x0000560b12e8bb58 in ExecProcNode (node=0x560b14176f40) at
> ../../../src/include/executor/executor.h:245
> #23 0x0000560b12e8bd59 in ExecSort (pstate=0x560b14176d28) at nodeSort.c:108
> #24 0x0000560b12e24e23 in ExecProcNodeFirst (node=0x560b14176d28) at
> execProcnode.c:450
> #25 0x0000560b12e10e71 in ExecProcNode (node=0x560b14176d28) at
> ../../../src/include/executor/executor.h:245
> #26 0x0000560b12e15c4c in ExecutePlan (estate=0x560b14176af0,
> planstate=0x560b14176d28, use_parallel_mode=false,
> operation=CMD_SELECT, sendTuples=true, numberTuples=0,
> direction=ForwardScanDirection, dest=0x560b1419d188,
> execute_once=true) at execMain.c:1646
> #27 0x0000560b12e11a19 in standard_ExecutorRun
> (queryDesc=0x560b1412db10, direction=ForwardScanDirection, count=0,
> execute_once=true) at execMain.c:364
> #28 0x0000560b12e116e1 in ExecutorRun (queryDesc=0x560b1412db10,
> direction=ForwardScanDirection, count=0, execute_once=true) at
> execMain.c:308
> #29 0x0000560b131f2177 in PortalRunSelect (portal=0x560b140db860,
> forward=true, count=0, dest=0x560b1419d188) at pquery.c:912
> #30 0x0000560b131f1b14 in PortalRun (portal=0x560b140db860,
> count=9223372036854775807, isTopLevel=true, run_once=true,
> dest=0x560b1419d188, altdest=0x560b1419d188,
> qc=0x7ffef18b2350) at pquery.c:756
> #31 0x0000560b131e550b in exec_simple_query (
> query_string=0x560b14076720 "/ display results, but hide most of the
> output /\nSELECT count(*), min(data), max(data)\nFROM
> pg_logical_slot_get_changes('regression_slot', NULL, NULL,
> 'include-xids', '0', 'skip-empty-xacts', '1')\nG"...) at
> postgres.c:1239
> #32 0x0000560b131ee343 in PostgresMain (argc=1, argv=0x560b1409faa0,
> dbname=0x560b1409f858 "contrib_regression", username=0x560b1409f830
> "mahendrathalor") at postgres.c:4315
> #33 0x0000560b130a325b in BackendRun (port=0x560b14096880) at postmaster.c:4510
> #34 0x0000560b130a22c3 in BackendStartup (port=0x560b14096880) at
> postmaster.c:4202
> #35 0x0000560b1309a5cc in ServerLoop () at postmaster.c:1727
> #36 0x0000560b130997c9 in PostmasterMain (argc=8, argv=0x560b1406f010)
> at postmaster.c:1400
> #37 0x0000560b12ee9530 in main (argc=8, argv=0x560b1406f010) at main.c:210
>
> I have Ubuntu setup. I think, this is reproducing into Ubuntu only. I
> am looking into this issue with Dilip.
This error is due to invalid size.
diff --git a/src/backend/replication/logical/reorderbuffer.c
b/src/backend/replication/logical/reorderbuffer.c
index eed9a5048b..487c1b4252 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -3678,7 +3678,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb,
ReorderBufferTXN *txn,
change->data.inval.invalidations =
MemoryContextAlloc(rb->context,
-
change->data.msg.message_size);
+
inval_size);
/* read the message */
memcpy(change->data.inval.invalidations, data, inval_size);
data += inval_size;
Above change, fixes the error. Thanks Dilip for helping.
--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-16 09:33 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-21 12:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 14:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 16:01 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-23 03:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-23 08:58 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Erik Rijkers <er@xs4all.nl>
2020-04-24 06:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Mahendra Singh Thalor <mahi6run@gmail.com>
2020-04-29 07:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Mahendra Singh Thalor <mahi6run@gmail.com>
@ 2020-04-29 07:20 ` Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-04-29 07:20 UTC (permalink / raw)
To: Mahendra Singh Thalor <mahi6run@gmail.com>; +Cc: Erik Rijkers <er@xs4all.nl>; Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Apr 29, 2020 at 12:37 PM Mahendra Singh Thalor
<mahi6run@gmail.com> wrote:
>
> On Wed, 29 Apr 2020 at 11:15, Mahendra Singh Thalor <mahi6run@gmail.com> wrote:
> >
> > On Fri, 24 Apr 2020 at 11:55, Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Thu, Apr 23, 2020 at 2:28 PM Erik Rijkers <er@xs4all.nl> wrote:
> > > >
> > > > On 2020-04-23 05:24, Dilip Kumar wrote:
> > > > > On Wed, Apr 22, 2020 at 9:31 PM Erik Rijkers <er@xs4all.nl> wrote:
> > > > >>
> > > > >> The 'ddl' one is apparently not quite fixed - I get this in (cd
> > > > >> contrib; make check)' (in both assert-enabled and non-assert-enabled
> > > > >> build)
> > > > >
> > > > > Can you send me the contrib/test_decoding/regression.diffs file?
> > > >
> > > > Attached.
> > >
> > > So from regression.diff, it appears that in failing in memory
> > > allocation (+ERROR: invalid memory alloc request size
> > > 94119198201896). My colleague tried to reproduce this in a different
> > > environment but there is no success so far. One more thing surprises
> > > me is that after
> > > (v15-0011-Provide-new-api-to-get-the-streaming-changes.patch)
> > > actually, it should never go for the streaming path. However, we can
> > > not ignore the fact that some of the changes might impact the
> > > non-streaming path as well. Is it possible for you to somehow stop or
> > > break the code and send the stack trace? One idea is by seeing the
> > > log we can see from where the error is raised i.e MemoryContextAlloc
> > > or palloc or some other similar function. Once we know that we can
> > > convert that error to an assert and find the call stack.
> > >
> > > --
> >
> > Thanks Erik for reporting this issue.
> >
> > I am able to reproduce this issue(+ERROR: invalid memory alloc
> > request size) on the top of v16 patch set. I applied all patches(12
> > patches) of v16 series and then I fired "make check -i" from
> > "contrib/test_decoding" folder. Below is stack trace of error:
> >
> > #0 0x0000560b1350902d in MemoryContextAlloc (context=0x560b14188d70,
> > size=94605581787992) at mcxt.c:806
> > #1 0x0000560b130f0ad5 in ReorderBufferRestoreChange
> > (rb=0x560b14188e90, txn=0x560b141baf08, data=0x560b1418a5e8 "K") at
> > reorderbuffer.c:3680
> > #2 0x0000560b130f0662 in ReorderBufferRestoreChanges
> > (rb=0x560b14188e90, txn=0x560b141baf08, file=0x560b1418ad10,
> > segno=0x560b1418ad20) at reorderbuffer.c:3564
> > #3 0x0000560b130e918a in ReorderBufferIterTXNInit (rb=0x560b14188e90,
> > txn=0x560b141baf08, iter_state=0x7ffef18b1600) at reorderbuffer.c:1186
> > #4 0x0000560b130eaee1 in ReorderBufferProcessTXN (rb=0x560b14188e90,
> > txn=0x560b141baf08, commit_lsn=25986584, snapshot_now=0x560b141b74d8,
> > command_id=0, streaming=false)
> > at reorderbuffer.c:1785
> > #5 0x0000560b130ecae1 in ReorderBufferCommit (rb=0x560b14188e90,
> > xid=508, commit_lsn=25986584, end_lsn=25989088,
> > commit_time=641449268431600, origin_id=0, origin_lsn=0)
> > at reorderbuffer.c:2315
> > #6 0x0000560b130d14a1 in DecodeCommit (ctx=0x560b1416ea80,
> > buf=0x7ffef18b19b0, parsed=0x7ffef18b1850, xid=508) at decode.c:654
> > #7 0x0000560b130cff98 in DecodeXactOp (ctx=0x560b1416ea80,
> > buf=0x7ffef18b19b0) at decode.c:261
> > #8 0x0000560b130cf99a in LogicalDecodingProcessRecord
> > (ctx=0x560b1416ea80, record=0x560b1416ee00) at decode.c:130
> > #9 0x0000560b130dbbbc in pg_logical_slot_get_changes_guts
> > (fcinfo=0x560b1417ee50, confirm=true, binary=false, streaming=false)
> > at logicalfuncs.c:285
> > #10 0x0000560b130dbe71 in pg_logical_slot_get_changes
> > (fcinfo=0x560b1417ee50) at logicalfuncs.c:354
> > #11 0x0000560b12e294d4 in ExecMakeTableFunctionResult
> > (setexpr=0x560b14177838, econtext=0x560b14177748,
> > argContext=0x560b1417ed30, expectedDesc=0x560b141814a0,
> > randomAccess=false) at execSRF.c:234
> > #12 0x0000560b12e5490f in FunctionNext (node=0x560b14177630) at
> > nodeFunctionscan.c:94
> > #13 0x0000560b12e2c108 in ExecScanFetch (node=0x560b14177630,
> > accessMtd=0x560b12e54836 <FunctionNext>, recheckMtd=0x560b12e54e15
> > <FunctionRecheck>) at execScan.c:133
> > #14 0x0000560b12e2c227 in ExecScan (node=0x560b14177630,
> > accessMtd=0x560b12e54836 <FunctionNext>, recheckMtd=0x560b12e54e15
> > <FunctionRecheck>) at execScan.c:199
> > #15 0x0000560b12e54e9b in ExecFunctionScan (pstate=0x560b14177630) at
> > nodeFunctionscan.c:270
> > #16 0x0000560b12e24e23 in ExecProcNodeFirst (node=0x560b14177630) at
> > execProcnode.c:450
> > #17 0x0000560b12e3e172 in ExecProcNode (node=0x560b14177630) at
> > ../../../src/include/executor/executor.h:245
> > #18 0x0000560b12e3e998 in fetch_input_tuple (aggstate=0x560b14176f40)
> > at nodeAgg.c:566
> > #19 0x0000560b12e4398f in agg_fill_hash_table
> > (aggstate=0x560b14176f40) at nodeAgg.c:2518
> > #20 0x0000560b12e42c9a in ExecAgg (pstate=0x560b14176f40) at nodeAgg.c:2139
> > #21 0x0000560b12e24e23 in ExecProcNodeFirst (node=0x560b14176f40) at
> > execProcnode.c:450
> > #22 0x0000560b12e8bb58 in ExecProcNode (node=0x560b14176f40) at
> > ../../../src/include/executor/executor.h:245
> > #23 0x0000560b12e8bd59 in ExecSort (pstate=0x560b14176d28) at nodeSort.c:108
> > #24 0x0000560b12e24e23 in ExecProcNodeFirst (node=0x560b14176d28) at
> > execProcnode.c:450
> > #25 0x0000560b12e10e71 in ExecProcNode (node=0x560b14176d28) at
> > ../../../src/include/executor/executor.h:245
> > #26 0x0000560b12e15c4c in ExecutePlan (estate=0x560b14176af0,
> > planstate=0x560b14176d28, use_parallel_mode=false,
> > operation=CMD_SELECT, sendTuples=true, numberTuples=0,
> > direction=ForwardScanDirection, dest=0x560b1419d188,
> > execute_once=true) at execMain.c:1646
> > #27 0x0000560b12e11a19 in standard_ExecutorRun
> > (queryDesc=0x560b1412db10, direction=ForwardScanDirection, count=0,
> > execute_once=true) at execMain.c:364
> > #28 0x0000560b12e116e1 in ExecutorRun (queryDesc=0x560b1412db10,
> > direction=ForwardScanDirection, count=0, execute_once=true) at
> > execMain.c:308
> > #29 0x0000560b131f2177 in PortalRunSelect (portal=0x560b140db860,
> > forward=true, count=0, dest=0x560b1419d188) at pquery.c:912
> > #30 0x0000560b131f1b14 in PortalRun (portal=0x560b140db860,
> > count=9223372036854775807, isTopLevel=true, run_once=true,
> > dest=0x560b1419d188, altdest=0x560b1419d188,
> > qc=0x7ffef18b2350) at pquery.c:756
> > #31 0x0000560b131e550b in exec_simple_query (
> > query_string=0x560b14076720 "/ display results, but hide most of the
> > output /\nSELECT count(*), min(data), max(data)\nFROM
> > pg_logical_slot_get_changes('regression_slot', NULL, NULL,
> > 'include-xids', '0', 'skip-empty-xacts', '1')\nG"...) at
> > postgres.c:1239
> > #32 0x0000560b131ee343 in PostgresMain (argc=1, argv=0x560b1409faa0,
> > dbname=0x560b1409f858 "contrib_regression", username=0x560b1409f830
> > "mahendrathalor") at postgres.c:4315
> > #33 0x0000560b130a325b in BackendRun (port=0x560b14096880) at postmaster.c:4510
> > #34 0x0000560b130a22c3 in BackendStartup (port=0x560b14096880) at
> > postmaster.c:4202
> > #35 0x0000560b1309a5cc in ServerLoop () at postmaster.c:1727
> > #36 0x0000560b130997c9 in PostmasterMain (argc=8, argv=0x560b1406f010)
> > at postmaster.c:1400
> > #37 0x0000560b12ee9530 in main (argc=8, argv=0x560b1406f010) at main.c:210
> >
> > I have Ubuntu setup. I think, this is reproducing into Ubuntu only. I
> > am looking into this issue with Dilip.
>
> This error is due to invalid size.
>
> diff --git a/src/backend/replication/logical/reorderbuffer.c
> b/src/backend/replication/logical/reorderbuffer.c
> index eed9a5048b..487c1b4252 100644
> --- a/src/backend/replication/logical/reorderbuffer.c
> +++ b/src/backend/replication/logical/reorderbuffer.c
> @@ -3678,7 +3678,7 @@ ReorderBufferRestoreChange(ReorderBuffer *rb,
> ReorderBufferTXN *txn,
>
> change->data.inval.invalidations =
> MemoryContextAlloc(rb->context,
> -
> change->data.msg.message_size);
> +
> inval_size);
> /* read the message */
>
> memcpy(change->data.inval.invalidations, data, inval_size);
> data += inval_size;
>
> Above change, fixes the error. Thanks Dilip for helping.
Thanks, Mahendra for reproducing and help in fixing this. I will
include this change in my next patch set.
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-16 20:10 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Kuntal Ghosh @ 2020-04-16 20:10 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Apr 14, 2020 at 3:41 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
Few review comments from 0006-Add-support-for-streaming*.patch
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
lseek can return (-)ve value in case of error, right?
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ *
+ * Don't check for error from mkdir; it could fail if the directory
+ * already exists (maybe someone else just did the same thing). If
+ * it doesn't work then we'll bomb out when opening the file
+ */
+ mkdir(tempdirpath, S_IRWXU);
If that's the only reason, perhaps we can use something like following:
if (mkdir(tempdirpath, S_IRWXU) < 0 && errno != EEXIST)
throw error;
+
+ CloseTransientFile(stream_fd);
Might failed to close the file. We should handle the case.
Also, I think we need some implementations in dumpSubscription() to
dump the (streaming = 'on') option.
--
Thanks & Regards,
Kuntal Ghosh
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
@ 2020-04-27 10:34 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-27 10:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-28 09:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-04-27 10:34 UTC (permalink / raw)
To: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Amit Kapila <amit.kapila16@gmail.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Fri, Apr 17, 2020 at 1:40 AM Kuntal Ghosh <kuntalghosh.2007@gmail.com> wrote:
>
> On Tue, Apr 14, 2020 at 3:41 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
>
> Few review comments from 0006-Add-support-for-streaming*.patch
>
> + subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
> lseek can return (-)ve value in case of error, right?
>
> + /*
> + * We might need to create the tablespace's tempfile directory, if no
> + * one has yet done so.
> + *
> + * Don't check for error from mkdir; it could fail if the directory
> + * already exists (maybe someone else just did the same thing). If
> + * it doesn't work then we'll bomb out when opening the file
> + */
> + mkdir(tempdirpath, S_IRWXU);
> If that's the only reason, perhaps we can use something like following:
>
> if (mkdir(tempdirpath, S_IRWXU) < 0 && errno != EEXIST)
> throw error;
Done
>
> +
> + CloseTransientFile(stream_fd);
> Might failed to close the file. We should handle the case.
Changed
Still, one place is pending because I don't have the filename there to
report an error. One option is we can just give an error without the
filename. I will try to think about this part.
> Also, I think we need some implementations in dumpSubscription() to
> dump the (streaming = 'on') option.
Right, created another patch and attached.
I have also fixed a couple of bugs internally reported by my colleague
Neha Sharma.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
Attachments:
v16-0002-Issue-individual-invalidations-with-wal_level-lo.patch (application/octet-stream, 16.7K) download
From 36577ba865b02113645c76c92b214b606a634727 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Tue, 14 Apr 2020 11:11:37 +0530
Subject: [PATCH v16 02/12] Issue individual invalidations with
wal_level=logical
When wal_level=logical, write individual invalidations into WAL so
that decoding can use this information.
We still add the invalidations to the cache, and write them to WAL
at commit time in RecordTransactionCommit(). This uses the existing
XLOG_INVALIDATIONS xlog record type, from the RM_STANDBY_ID resource
manager (see LogStandbyInvalidations for details).
So existing code relying on those invalidations (e.g. redo) does not
need to be changed.
The individual invalidations are written are written using a new
xlog record type XLOG_XACT_INVALIDATIONS, from RM_XACT_ID resource
manager. See LogLogicalInvalidations for details.
These new xlog records are ignored by existing redo procedures,
which still rely on the invalidations written to commit records.
The invalidations are decoded and added as a new ReorderBufferChange
type (REORDER_BUFFER_CHANGE_INVALIDATION), and then executed during
replay, unlike the existing invalidations (which are either decoded
as part of commit record, or executed immediately during decoding
and not added to reorderbuffer at all).
LogStandbyInvalidations was accumulating all the invalidations in
memory, and then only wrote them once at commit time, which may
reduce the performance impact by amortizing the overhead and
deduplicating the invalidations.
---
src/backend/access/rmgrdesc/xactdesc.c | 40 +++++++++
src/backend/access/transam/xact.c | 7 ++
src/backend/replication/logical/decode.c | 16 ++++
src/backend/replication/logical/reorderbuffer.c | 104 +++++++++++++++++++++---
src/backend/utils/cache/inval.c | 49 +++++++++++
src/include/access/xact.h | 13 ++-
src/include/replication/reorderbuffer.h | 11 +++
7 files changed, 226 insertions(+), 14 deletions(-)
diff --git a/src/backend/access/rmgrdesc/xactdesc.c b/src/backend/access/rmgrdesc/xactdesc.c
index fbc5942..17c06f7 100644
--- a/src/backend/access/rmgrdesc/xactdesc.c
+++ b/src/backend/access/rmgrdesc/xactdesc.c
@@ -20,6 +20,9 @@
#include "storage/standbydefs.h"
#include "utils/timestamp.h"
+static void xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs);
+
/*
* Parse the WAL format of an xact commit and abort records into an easier to
* understand format.
@@ -396,6 +399,12 @@ xact_desc(StringInfo buf, XLogReaderState *record)
appendStringInfo(buf, "xtop %u: ", xlrec->xtop);
xact_desc_assignment(buf, xlrec);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ xl_xact_invalidations *xlrec = (xl_xact_invalidations *) rec;
+
+ xact_desc_invalidations(buf, xlrec->nmsgs, xlrec->msgs);
+ }
}
const char *
@@ -423,7 +432,38 @@ xact_identify(uint8 info)
case XLOG_XACT_ASSIGNMENT:
id = "ASSIGNMENT";
break;
+ case XLOG_XACT_INVALIDATIONS:
+ id = "INVALIDATION";
+ break;
}
return id;
}
+
+static void
+xact_desc_invalidations(StringInfo buf,
+ int nmsgs, SharedInvalidationMessage *msgs)
+{
+ int i;
+
+ appendStringInfoString(buf, "; inval msgs:");
+ for (i = 0; i < nmsgs; i++)
+ {
+ SharedInvalidationMessage *msg = &msgs[i];
+
+ if (msg->id >= 0)
+ appendStringInfo(buf, " catcache %d", msg->id);
+ else if (msg->id == SHAREDINVALCATALOG_ID)
+ appendStringInfo(buf, " catalog %u", msg->cat.catId);
+ else if (msg->id == SHAREDINVALRELCACHE_ID)
+ appendStringInfo(buf, " relcache %u", msg->rc.relId);
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALSMGR_ID)
+ appendStringInfoString(buf, " smgr");
+ /* not expected, but print something anyway */
+ else if (msg->id == SHAREDINVALRELMAP_ID)
+ appendStringInfo(buf, " relmap db %u", msg->rm.dbId);
+ else
+ appendStringInfo(buf, " unrecognized id %d", msg->id);
+ }
+}
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index c2604bb..8e6b1a6 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -6020,6 +6020,13 @@ xact_redo(XLogReaderState *record)
ProcArrayApplyXidAssignment(xlrec->xtop,
xlrec->nsubxacts, xlrec->xsub);
}
+ else if (info == XLOG_XACT_INVALIDATIONS)
+ {
+ /*
+ * XXX we do ignore this for now, what matters are invalidations
+ * written into the commit record.
+ */
+ }
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 122c581..69c1f45 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -281,6 +281,22 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
}
case XLOG_XACT_ASSIGNMENT:
break;
+ case XLOG_XACT_INVALIDATIONS:
+ {
+ TransactionId xid;
+ xl_xact_invalidations *invals;
+
+ xid = XLogRecGetXid(r);
+ invals = (xl_xact_invalidations *) XLogRecGetData(r);
+
+ Assert(TransactionIdIsValid(xid));
+ ReorderBufferAddInvalidation(reorder, xid, buf->origptr,
+ invals->nmsgs, invals->msgs);
+
+
+ ReorderBufferXidSetCatalogChanges(ctx->reorder, xid, buf->origptr);
+ }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 4594cf9..0d5bb73 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -220,7 +220,7 @@ static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
-static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs);
/*
* ---------------------------------------
@@ -455,6 +455,11 @@ ReorderBufferReturnChange(ReorderBuffer *rb, ReorderBufferChange *change)
pfree(change->data.msg.message);
change->data.msg.message = NULL;
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ if (change->data.inval.invalidations)
+ pfree(change->data.inval.invalidations);
+ change->data.inval.invalidations = NULL;
+ break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
if (change->data.snapshot)
{
@@ -1814,17 +1819,24 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
TeardownHistoricSnapshot(false);
SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
-
- /*
- * Every time the CommandId is incremented, we could
- * see new catalog contents, so execute all
- * invalidations.
- */
- ReorderBufferExecuteInvalidations(rb, txn);
}
break;
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+
+ /*
+ * Execute the invalidation messages locally.
+ *
+ * XXX Do we need to care about relcacheInitFileInval and
+ * the other fields added to ReorderBufferChange, or just
+ * about the message itself?
+ */
+ ReorderBufferExecuteInvalidations(
+ change->data.inval.ninvalidations,
+ change->data.inval.invalidations);
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
elog(ERROR, "tuplecid value in changequeue");
break;
@@ -1866,7 +1878,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -1892,7 +1905,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2204,6 +2218,33 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
/*
* Setup the invalidation of the toplevel transaction.
+ */
+void
+ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, int nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ MemoryContext oldcontext;
+ ReorderBufferChange *change;
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ MemoryContextAlloc(rb->context,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Setup the invalidation of the toplevel transaction.
*
* This needs to be done before ReorderBufferCommit is called!
*/
@@ -2234,12 +2275,12 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
* in the changestream but we don't know which those are.
*/
static void
-ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs)
{
int i;
- for (i = 0; i < txn->ninvalidations; i++)
- LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+ for (i = 0; i < nmsgs; i++)
+ LocalExecuteInvalidationMessage(&msgs[i]);
}
/*
@@ -2591,6 +2632,24 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ {
+ char *data;
+ Size inval_size = sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+
+ sz += inval_size;
+
+ ReorderBufferSerializeReserve(rb, sz);
+ data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange);
+
+ /* might have been reallocated above */
+ ondisk = (ReorderBufferDiskChange *) rb->outbuf;
+ memcpy(data, change->data.inval.invalidations, inval_size);
+ data += inval_size;
+
+ break;
+ }
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot snap;
@@ -2736,6 +2795,11 @@ ReorderBufferChangeSize(ReorderBufferChange *change)
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ sz += sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+ break;
+
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot snap;
@@ -3004,6 +3068,20 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
break;
}
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ {
+ Size inval_size = sizeof(SharedInvalidationMessage) *
+ change->data.inval.ninvalidations;
+
+ change->data.inval.invalidations =
+ MemoryContextAlloc(rb->context,
+ change->data.msg.message_size);
+ /* read the message */
+ memcpy(change->data.inval.invalidations, data, inval_size);
+ data += inval_size;
+
+ break;
+ }
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
{
Snapshot oldsnap;
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 591dd33..cba5b6c 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -85,6 +85,12 @@
* worth trying to avoid sending such inval traffic in the future, if those
* problems can be overcome cheaply.
*
+ * When wal_level=logical, write invalidations into WAL at each command end to
+ * support the decoding of the in-progress transaction. As of now it was
+ * enough to log invalidation only at commit because we are only decoding the
+ * transaction at the commit time. We only need to log the catalog cache and
+ * relcache invalidation. There can not be any active MVCC scan in logical
+ * decoding so we don't need to log the snapshot invalidation.
*
* Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,6 +110,7 @@
#include "catalog/pg_constraint.h"
#include "miscadmin.h"
#include "storage/sinval.h"
+#include "storage/standby.h"
#include "storage/smgr.h"
#include "utils/catcache.h"
#include "utils/inval.h"
@@ -210,6 +217,8 @@ static struct RELCACHECALLBACK
static int relcache_callback_count = 0;
+static void LogLogicalInvalidations(void);
+
/* ----------------------------------------------------------------
* Invalidation list support functions
*
@@ -1092,6 +1101,9 @@ CommandEndInvalidationMessages(void)
if (transInvalInfo == NULL)
return;
+ if (XLogLogicalInfoActive())
+ LogLogicalInvalidations();
+
ProcessInvalidationMessages(&transInvalInfo->CurrentCmdInvalidMsgs,
LocalExecuteInvalidationMessage);
AppendInvalidationMessages(&transInvalInfo->PriorCmdInvalidMsgs,
@@ -1501,3 +1513,40 @@ CallSyscacheCallbacks(int cacheid, uint32 hashvalue)
i = ccitem->link - 1;
}
}
+
+/*
+ * Emit WAL for invalidations.
+ */
+static void
+LogLogicalInvalidations()
+{
+ xl_xact_invalidations xlrec;
+ SharedInvalidationMessage *invalMessages;
+ int nmsgs = 0;
+
+ if (transInvalInfo->CurrentCmdInvalidMsgs.cclist)
+ {
+ ProcessInvalidationMessagesMulti(&transInvalInfo->CurrentCmdInvalidMsgs,
+ MakeSharedInvalidMessagesArray);
+ invalMessages = SharedInvalidMessagesArray;
+ nmsgs = numSharedInvalidMessagesArray;
+ SharedInvalidMessagesArray = NULL;
+ numSharedInvalidMessagesArray = 0;
+ }
+
+ if (nmsgs > 0)
+ {
+ /* prepare record */
+ memset(&xlrec, 0, MinSizeOfXactInvalidations);
+ xlrec.nmsgs = nmsgs;
+
+ /* perform insertion */
+ XLogBeginInsert();
+ XLogRegisterData((char *) (&xlrec), MinSizeOfXactInvalidations);
+ XLogRegisterData((char *) invalMessages,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ XLogInsert(RM_XACT_ID, XLOG_XACT_INVALIDATIONS);
+
+ pfree(invalMessages);
+ }
+}
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 8645b38..b822c5e 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -146,7 +146,7 @@ typedef void (*SubXactCallback) (SubXactEvent event, SubTransactionId mySubid,
#define XLOG_XACT_COMMIT_PREPARED 0x30
#define XLOG_XACT_ABORT_PREPARED 0x40
#define XLOG_XACT_ASSIGNMENT 0x50
-/* free opcode 0x60 */
+#define XLOG_XACT_INVALIDATIONS 0x60
/* free opcode 0x70 */
/* mask for filtering opcodes out of xl_info */
@@ -198,6 +198,17 @@ typedef struct xl_xact_assignment
#define MinSizeOfXactAssignment offsetof(xl_xact_assignment, xsub)
/*
+ * Invalidations logged with wal_level=logical.
+ */
+typedef struct xl_xact_invalidations
+{
+ int nmsgs; /* number of shared inval msgs */
+ SharedInvalidationMessage msgs[FLEXIBLE_ARRAY_MEMBER];
+} xl_xact_invalidations;
+
+#define MinSizeOfXactInvalidations offsetof(xl_xact_invalidations, msgs)
+
+/*
* Commit and abort records can contain a lot of information. But a large
* portion of the records won't need all possible pieces of information. So we
* only include what's needed.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 626ecf4..af35287 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -57,6 +57,7 @@ enum ReorderBufferChangeType
REORDER_BUFFER_CHANGE_UPDATE,
REORDER_BUFFER_CHANGE_DELETE,
REORDER_BUFFER_CHANGE_MESSAGE,
+ REORDER_BUFFER_CHANGE_INVALIDATION,
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT,
REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID,
REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID,
@@ -149,6 +150,14 @@ typedef struct ReorderBufferChange
CommandId cmax;
CommandId combocid;
} tuplecid;
+
+ /* Invalidation. */
+ struct
+ {
+ uint32 ninvalidations; /* Number of messages */
+ SharedInvalidationMessage *invalidations; /* invalidation
+ * message */
+ } inval;
} data;
/*
@@ -459,6 +468,8 @@ void ReorderBufferAddNewCommandId(ReorderBuffer *, TransactionId, XLogRecPtr ls
void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
RelFileNode node, ItemPointerData pt,
CommandId cmin, CommandId cmax, CommandId combocid);
+void ReorderBufferAddInvalidation(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
+ int nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
--
1.8.3.1
v16-0001-Immediately-WAL-log-assignments.patch (application/octet-stream, 10.5K) download
From e856c586bb131e1047471ecfddc6b0f118f1b0f3 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 20 Mar 2020 15:03:01 +0530
Subject: [PATCH v16 01/12] Immediately WAL-log assignments
The logical decoding infrastructure needs to know which top-level
transaction the subxact belongs to, in order to decode all the
changes. Until now that might be delayed until commit, due to the
caching (GPROC_MAX_CACHED_SUBXIDS), preventing features requiring
incremental decoding.
So we also write the assignment info into WAL immediately, as part
of the next WAL record (to minimize overhead). However, we can not
remove the existing XLOG_XACT_ASSIGNMENT wal as that is required
for avoiding overflow in the hot standby snapshot.
---
src/backend/access/transam/xact.c | 46 ++++++++++++++++++++++++++++++++
src/backend/access/transam/xloginsert.c | 22 +++++++++++++--
src/backend/access/transam/xlogreader.c | 5 ++++
src/backend/replication/logical/decode.c | 39 ++++++++++++++-------------
src/include/access/xact.h | 3 +++
src/include/access/xlog.h | 1 +
src/include/access/xlogreader.h | 3 +++
src/include/access/xlogrecord.h | 1 +
8 files changed, 99 insertions(+), 21 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 3984dd3..c2604bb 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -191,6 +191,7 @@ typedef struct TransactionStateData
bool didLogXid; /* has xid been included in WAL record? */
int parallelModeLevel; /* Enter/ExitParallelMode counter */
bool chain; /* start a new block after this one */
+ bool assigned; /* assigned to toplevel XID */
struct TransactionStateData *parent; /* back link to parent */
} TransactionStateData;
@@ -223,6 +224,7 @@ typedef struct SerializedTransactionState
static TransactionStateData TopTransactionStateData = {
.state = TRANS_DEFAULT,
.blockState = TBLOCK_DEFAULT,
+ .assigned = false,
};
/*
@@ -5118,6 +5120,7 @@ PushTransaction(void)
GetUserIdAndSecContext(&s->prevUser, &s->prevSecContext);
s->prevXactReadOnly = XactReadOnly;
s->parallelModeLevel = 0;
+ s->assigned = false;
CurrentTransactionState = s;
@@ -6020,3 +6023,46 @@ xact_redo(XLogReaderState *record)
else
elog(PANIC, "xact_redo: unknown op code %u", info);
}
+
+/*
+ * IsSubTransactionAssignmentPending
+ *
+ * This returns true if we are inside a valid substransaction, for which
+ * the assignment was not yet written to any WAL record.
+ */
+bool
+IsSubTransactionAssignmentPending(void)
+{
+ if (!XLogLogicalInfoActive())
+ return false;
+
+ /* we need to be in a transaction state */
+ if (!IsTransactionState())
+ return false;
+
+ /* it has to be a subtransaction */
+ if (!IsSubTransaction())
+ return false;
+
+ /* the subtransaction has to have a XID assigned */
+ if (!TransactionIdIsValid(GetCurrentTransactionIdIfAny()))
+ return false;
+
+ /* and it needs to have 'assigned' */
+ return !CurrentTransactionState->assigned;
+
+}
+
+/*
+ * MarkSubTransactionAssigned
+ *
+ * Mark the subtransaction assignment as completed.
+ */
+void
+MarkSubTransactionAssigned(void)
+{
+ Assert(IsSubTransactionAssignmentPending());
+
+ CurrentTransactionState->assigned = true;
+
+}
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 4259309..3c49954 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -89,11 +89,13 @@ static XLogRecData hdr_rdt;
static char *hdr_scratch = NULL;
#define SizeOfXlogOrigin (sizeof(RepOriginId) + sizeof(char))
+#define SizeOfXLogTransactionId (sizeof(TransactionId) + sizeof(char))
#define HEADER_SCRATCH_SIZE \
(SizeOfXLogRecord + \
MaxSizeOfXLogRecordBlockHeader * (XLR_MAX_BLOCK_ID + 1) + \
- SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin)
+ SizeOfXLogRecordDataHeaderLong + SizeOfXlogOrigin + \
+ SizeOfXLogTransactionId)
/*
* An array of XLogRecData structs, to hold registered data.
@@ -195,6 +197,10 @@ XLogResetInsertion(void)
{
int i;
+ /* reset the subxact assignment flag (if needed) */
+ if (curinsert_flags & XLOG_INCLUDE_XID)
+ MarkSubTransactionAssigned();
+
for (i = 0; i < max_registered_block_id; i++)
registered_buffers[i].in_use = false;
@@ -398,7 +404,7 @@ void
XLogSetRecordFlags(uint8 flags)
{
Assert(begininsert_called);
- curinsert_flags = flags;
+ curinsert_flags |= flags;
}
/*
@@ -748,6 +754,18 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
scratch += sizeof(replorigin_session_origin);
}
+ /* followed by toplevel XID, if not already included in previous record */
+ if (IsSubTransactionAssignmentPending())
+ {
+ TransactionId xid = GetTopTransactionIdIfAny();
+
+ /* update the flag (later used by XLogInsertRecord) */
+ XLogSetRecordFlags(XLOG_INCLUDE_XID);
+ *(scratch++) = (char) XLR_BLOCK_ID_TOPLEVEL_XID;
+ memcpy(scratch, &xid, sizeof(TransactionId));
+ scratch += sizeof(TransactionId);
+ }
+
/* followed by main data, if any */
if (mainrdata_len > 0)
{
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 79ff976..7b5257f 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1189,6 +1189,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
state->decoded_record = record;
state->record_origin = InvalidRepOriginId;
+ state->toplevel_xid = InvalidTransactionId;
ptr = (char *) record;
ptr += SizeOfXLogRecord;
@@ -1227,6 +1228,10 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
{
COPY_HEADER_FIELD(&state->record_origin, sizeof(RepOriginId));
}
+ else if (block_id == XLR_BLOCK_ID_TOPLEVEL_XID)
+ {
+ COPY_HEADER_FIELD(&state->toplevel_xid, sizeof(TransactionId));
+ }
else if (block_id <= XLR_MAX_BLOCK_ID)
{
/* XLogRecordBlockHeader */
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index c2e5e3a..122c581 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -93,12 +93,28 @@ static void DecodeXLogTuple(char *data, Size len, ReorderBufferTupleBuf *tup);
void
LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *record)
{
- XLogRecordBuffer buf;
+ XLogRecordBuffer buf;
+ TransactionId txid;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
buf.record = record;
+ txid = XLogRecGetTopXid(record);
+
+ /*
+ * If the toplevel xid is valid, we need to assign the subxact to the
+ * toplevel xact. We need to do this for all records, hence we do it before
+ * the switch.
+ */
+ if (TransactionIdIsValid(txid))
+ {
+ ReorderBufferAssignChild(ctx->reorder,
+ txid,
+ record->decoded_record->xl_xid,
+ buf.origptr);
+ }
+
/* cast so we get a warning when new rmgrs are added */
switch ((RmgrId) XLogRecGetRmid(record))
{
@@ -217,12 +233,12 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
* If the snapshot isn't yet fully built, we cannot decode anything, so
* bail out.
*
- * However, it's critical to process XLOG_XACT_ASSIGNMENT records even
+ * However, it's critical to process records with subxid assignment even
* when the snapshot is being built: it is possible to get later records
* that require subxids to be properly assigned.
*/
if (SnapBuildCurrentState(builder) < SNAPBUILD_FULL_SNAPSHOT &&
- info != XLOG_XACT_ASSIGNMENT)
+ !TransactionIdIsValid(XLogRecGetTopXid(r)))
return;
switch (info)
@@ -264,22 +280,7 @@ DecodeXactOp(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
break;
}
case XLOG_XACT_ASSIGNMENT:
- {
- xl_xact_assignment *xlrec;
- int i;
- TransactionId *sub_xid;
-
- xlrec = (xl_xact_assignment *) XLogRecGetData(r);
-
- sub_xid = &xlrec->xsub[0];
-
- for (i = 0; i < xlrec->nsubxacts; i++)
- {
- ReorderBufferAssignChild(reorder, xlrec->xtop,
- *(sub_xid++), buf->origptr);
- }
- break;
- }
+ break;
case XLOG_XACT_PREPARE:
/*
diff --git a/src/include/access/xact.h b/src/include/access/xact.h
index 7ee04ba..8645b38 100644
--- a/src/include/access/xact.h
+++ b/src/include/access/xact.h
@@ -428,6 +428,9 @@ extern void UnregisterXactCallback(XactCallback callback, void *arg);
extern void RegisterSubXactCallback(SubXactCallback callback, void *arg);
extern void UnregisterSubXactCallback(SubXactCallback callback, void *arg);
+extern bool IsSubTransactionAssignmentPending(void);
+extern void MarkSubTransactionAssigned(void);
+
extern int xactGetCommittedChildren(TransactionId **ptr);
extern XLogRecPtr XactLogCommitRecord(TimestampTz commit_time,
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index f60ed2d..6d439d0 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -229,6 +229,7 @@ extern bool XLOG_DEBUG;
*/
#define XLOG_INCLUDE_ORIGIN 0x01 /* include the replication origin */
#define XLOG_MARK_UNIMPORTANT 0x02 /* record not important for durability */
+#define XLOG_INCLUDE_XID 0x04 /* include XID of toplevel xact */
/* Checkpoint statistics */
diff --git a/src/include/access/xlogreader.h b/src/include/access/xlogreader.h
index 4582196..756f6df 100644
--- a/src/include/access/xlogreader.h
+++ b/src/include/access/xlogreader.h
@@ -150,6 +150,8 @@ struct XLogReaderState
RepOriginId record_origin;
+ TransactionId toplevel_xid; /* XID of toplevel transaction */
+
/* information about blocks referenced by the record. */
DecodedBkpBlock blocks[XLR_MAX_BLOCK_ID + 1];
@@ -285,6 +287,7 @@ extern bool DecodeXLogRecord(XLogReaderState *state, XLogRecord *record,
#define XLogRecGetRmid(decoder) ((decoder)->decoded_record->xl_rmid)
#define XLogRecGetXid(decoder) ((decoder)->decoded_record->xl_xid)
#define XLogRecGetOrigin(decoder) ((decoder)->record_origin)
+#define XLogRecGetTopXid(decoder) ((decoder)->toplevel_xid)
#define XLogRecGetData(decoder) ((decoder)->main_data)
#define XLogRecGetDataLen(decoder) ((decoder)->main_data_len)
#define XLogRecHasAnyBlockRefs(decoder) ((decoder)->max_block_id >= 0)
diff --git a/src/include/access/xlogrecord.h b/src/include/access/xlogrecord.h
index acd9af0..2f0c8bf 100644
--- a/src/include/access/xlogrecord.h
+++ b/src/include/access/xlogrecord.h
@@ -223,5 +223,6 @@ typedef struct XLogRecordDataHeaderLong
#define XLR_BLOCK_ID_DATA_SHORT 255
#define XLR_BLOCK_ID_DATA_LONG 254
#define XLR_BLOCK_ID_ORIGIN 253
+#define XLR_BLOCK_ID_TOPLEVEL_XID 252
#endif /* XLOGRECORD_H */
--
1.8.3.1
v16-0003-Extend-the-output-plugin-API-with-stream-methods.patch (application/octet-stream, 34.8K) download
From a65eda69a3bd6810223ea321d24f8080451e3351 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 17:26:31 +0200
Subject: [PATCH v16 03/12] Extend the output plugin API with stream methods
This adds four methods the output plugin API, adding support
for streaming changes for large transactions.
* stream_message
* stream_change
* stream_truncate
* stream_abort
* stream_commit
* stream_start
* stream_stop
Most of this is a simple extension of the existing methods, with
the semantic difference that the transaction (or subtransaction)
is incomplete and may be aborted later (which is something the
regular API does not really need to deal with).
This also extends the 'test_decoding' plugin, implementing these
new stream methods.
The stream_start/start_stop are used to demarcate the a chunk
of changes streamed for a particular toplevel transaction.
---
contrib/test_decoding/test_decoding.c | 100 +++++++++
doc/src/sgml/logicaldecoding.sgml | 209 +++++++++++++++++
src/backend/replication/logical/logical.c | 361 ++++++++++++++++++++++++++++++
src/include/replication/logical.h | 5 +
src/include/replication/output_plugin.h | 69 ++++++
src/include/replication/reorderbuffer.h | 57 +++++
6 files changed, 801 insertions(+)
diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c
index 93c9488..64f651f 100644
--- a/contrib/test_decoding/test_decoding.c
+++ b/contrib/test_decoding/test_decoding.c
@@ -62,6 +62,28 @@ static void pg_decode_message(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn, XLogRecPtr message_lsn,
bool transactional, const char *prefix,
Size sz, const char *message);
+static void pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr message_lsn,
+ bool transactional, const char *prefix,
+ Size sz, const char *message);
+static void pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+static void pg_decode_stream_truncate(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change);
+static void pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+static void pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
void
_PG_init(void)
@@ -83,6 +105,13 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->filter_by_origin_cb = pg_decode_filter;
cb->shutdown_cb = pg_decode_shutdown;
cb->message_cb = pg_decode_message;
+ cb->stream_message_cb = pg_decode_stream_message;
+ cb->stream_change_cb = pg_decode_stream_change;
+ cb->stream_truncate_cb = pg_decode_stream_truncate;
+ cb->stream_abort_cb = pg_decode_stream_abort;
+ cb->stream_commit_cb = pg_decode_stream_commit;
+ cb->stream_start_cb = pg_decode_stream_start;
+ cb->stream_stop_cb = pg_decode_stream_stop;
}
@@ -540,3 +569,74 @@ pg_decode_message(LogicalDecodingContext *ctx,
appendBinaryStringInfo(ctx->out, message, sz);
OutputPluginWrite(ctx, true);
}
+
+static void
+pg_decode_stream_message(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional,
+ const char *prefix, Size sz, const char *message)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:",
+ transactional, prefix, sz);
+ appendBinaryStringInfo(ctx->out, message, sz);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_change(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming change for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "streaming truncate for TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_abort(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "aborting streamed (sub)transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_commit(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "committing streamed transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_start(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "opening a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
+
+static void
+pg_decode_stream_stop(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ OutputPluginPrepareWrite(ctx, true);
+ appendStringInfo(ctx->out, "closing a streamed block for transaction TXN %u", txn->xid);
+ OutputPluginWrite(ctx, true);
+}
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index bad3bfe..65244b1 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -388,6 +388,13 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
@@ -400,6 +407,15 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
If <function>truncate_cb</function> is not set but a
<command>TRUNCATE</command> is to be decoded, the action will be ignored.
</para>
+
+ <para>
+ An output plugin may also define functions to support streaming of large,
+ in-progress transactions. The <function>stream_change_cb</function>,
+ <function>stream_commit_cb</function>, <function>stream_abort_cb</function>,
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function>
+ are required, while <function>stream_message_cb</function> and
+ <function>stream_truncate_cb</function> are optional.
+ </para>
</sect2>
<sect2 id="logicaldecoding-capabilities">
@@ -678,6 +694,112 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx,
</para>
</sect3>
+ <sect3 id="logicaldecoding-output-plugin-stream-start">
+ <title>Stream Start Callback</title>
+ <para>
+ The <function>stream_start_cb</function> callback is called when opening
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-stop">
+ <title>Stream Stop Callback</title>
+ <para>
+ The <function>stream_stop_cb</function> callback is called when closing
+ a block of streamed changes from an in-progress transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-change">
+ <title>Stream Change Callback</title>
+ <para>
+ The <function>stream_change_cb</function> callback is called when sending
+ a change in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-truncate">
+ <title>Stream Truncate Callback</title>
+ <para>
+ The <function>stream_truncate_cb</function> callback is called for a
+ <command>TRUNCATE</command> command in a block of streamed changes
+ (demarcated by <function>stream_start_cb</function> and
+ <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+</programlisting>
+ The parameters are analogous to the <function>stream_change_cb</function>
+ callback. However, because <command>TRUNCATE</command> actions on
+ tables connected by foreign keys need to be executed together, this
+ callback receives an array of relations instead of just a single one.
+ See the description of the <xref linkend="sql-truncate"/> statement for
+ details.
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-message">
+ <title>Stream Message Callback</title>
+ <para>
+ The <function>stream_message_cb</function> callback is called when sending
+ a generic message in a block of streamed changes (demarcated by
+ <function>stream_start_cb</function> and <function>stream_stop_cb</function> calls).
+<programlisting>
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-commit">
+ <title>Stream Commit Callback</title>
+ <para>
+ The <function>stream_commit_cb</function> callback is called to commit
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
+ <sect3 id="logicaldecoding-output-plugin-stream-abort">
+ <title>Stream Abort Callback</title>
+ <para>
+ The <function>stream_abort_cb</function> callback is called to abort
+ a previously streamed transaction.
+<programlisting>
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+</programlisting>
+ </para>
+ </sect3>
+
</sect2>
<sect2 id="logicaldecoding-output-plugin-output">
@@ -746,4 +868,91 @@ OutputPluginWrite(ctx, true);
</para>
</note>
</sect1>
+
+ <sect1 id="logicaldecoding-streaming">
+ <title>Streaming of Large Transactions for Logical Decoding</title>
+
+ <para>
+ The basic output plugin callbacks (e.g. <function>begin_cb</function>,
+ <function>change_cb</function>, <function>commit_cb</function> and
+ <function>message_cb</function>) are only invoked when the transaction
+ actually commits. The changes are still decoded from the transaction
+ log, but are only passed to the output plugin at commit (and discarded
+ if the transaction aborts).
+ </para>
+
+ <para>
+ This means that while the decoding happens incrementally, and may spill
+ to disk to keep memory usage under control, all the decoded changes have
+ to be transmitted when the transaction finally commits (or more precisely,
+ when the commit is decoded from the transaction log). Depending on the
+ size of the transaction and network bandwidth, the transfer time may
+ significantly increase the apply lag.
+ </para>
+
+ <para>
+ To reduce the apply lag caused by large transactions, an output plugin
+ may provide additional callback to support incremental streaming of
+ in-progress transactions. There are multiple required streaming callbacks
+ (<function>stream_change_cb</function>, <function>stream_commit_cb</function>,
+ <function>stream_abort_cb</function>, <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function>) and one optional callback
+ (<function>stream_message_cb</function>).
+ </para>
+
+ <para>
+ When streaming an in-progress transaction, the changes (and messages) are
+ streamed in blocks demarcated by <function>stream_start_cb</function>
+ and <function>stream_stop_cb</function> callbacks. Once all the decoded
+ changes are transmitted, the transaction is committed using the
+ <function>stream_commit_cb</function> callback (or possibly aborted using
+ the <function>stream_abort_cb</function> callback).
+ </para>
+
+ <para>
+ One example sequence of streaming callback calls for one transaction may
+ look like this:
+<programlisting>
+stream_start_cb(...); <-- start of first block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_message_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of first block of changes
+
+stream_start_cb(...); <-- start of second block of changes
+ stream_change_cb(...);
+ stream_change_cb(...);
+ stream_change_cb(...);
+ ...
+ stream_message_cb(...);
+ stream_change_cb(...);
+stream_stop_cb(...); <-- end of second block of changes
+
+stream_commit_cb(...); <-- commit of the streamed transaction
+</programlisting>
+ </para>
+
+ <para>
+ The actual sequence of callback calls may be more complicated, of course.
+ There may be blocks for multiple streamed transactions, some of the
+ transactions may get aborted, etc.
+ </para>
+
+ <para>
+ Similar to spill-to-disk behavior, streaming is triggered when the total
+ amount of changes decoded from the WAL (for all in-progress transactions)
+ exceeds limit defined by <varname>logical_decoding_work_mem</varname> setting.
+ At that point the largest toplevel transaction (measured by amount of memory
+ currently used for decoded changes) is selected and streamed.
+ </para>
+
+ <para>
+ Even when streaming large transactions, the changes are still applied in
+ commit order, preserving the same guarantees as the non-streaming mode.
+ </para>
+
+ </sect1>
</chapter>
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 5adf253..497d8a9 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -65,6 +65,21 @@ static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
XLogRecPtr message_lsn, bool transactional,
const char *prefix, Size message_size, const char *message);
+/* streaming callbacks */
+static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change);
+static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[], ReorderBufferChange *change);
+static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message);
+static void stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn);
+static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+static void stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn);
+
static void LoadOutputPlugin(OutputPluginCallbacks *callbacks, char *plugin);
/*
@@ -189,6 +204,39 @@ StartupDecodingContext(List *output_plugin_options,
ctx->reorder->commit = commit_cb_wrapper;
ctx->reorder->message = message_cb_wrapper;
+ /*
+ * To support streaming, we require change/commit/abort callbacks. The
+ * message callback is optional, similarly to regular output plugins. We
+ * however enable streaming when at least one of the methods is enabled,
+ * so that we can easily identify missing methods.
+ *
+ * We decide it here, but only check it later in the wrappers.
+ */
+ ctx->streaming = (ctx->callbacks.stream_change_cb != NULL) ||
+ (ctx->callbacks.stream_abort_cb != NULL) ||
+ (ctx->callbacks.stream_message_cb != NULL) ||
+ (ctx->callbacks.stream_truncate_cb != NULL) ||
+ (ctx->callbacks.stream_commit_cb != NULL) ||
+ (ctx->callbacks.stream_start_cb != NULL) ||
+ (ctx->callbacks.stream_stop_cb != NULL);
+
+ /*
+ * streaming callbacks
+ *
+ * stream_message and stream_truncate callbacks are optional,
+ * so we do not fail with ERROR when missing, but the wrappers
+ * simply do nothing. We must set the ReorderBuffer callbacks
+ * to something, otherwise the calls from there will crash (we
+ * don't want to move the checks there).
+ */
+ ctx->reorder->stream_change = stream_change_cb_wrapper;
+ ctx->reorder->stream_abort = stream_abort_cb_wrapper;
+ ctx->reorder->stream_commit = stream_commit_cb_wrapper;
+ ctx->reorder->stream_start = stream_start_cb_wrapper;
+ ctx->reorder->stream_stop = stream_stop_cb_wrapper;
+ ctx->reorder->stream_message = stream_message_cb_wrapper;
+ ctx->reorder->stream_truncate = stream_truncate_cb_wrapper;
+
ctx->out = makeStringInfo();
ctx->prepare_write = prepare_write;
ctx->write = do_write;
@@ -862,6 +910,319 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
error_context_stack = errcallback.previous;
}
+static void
+stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ Relation relation, ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_change";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ /* in streaming mode, stream_change_cb is required */
+ if (ctx->callbacks.stream_change_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_change_cb callback.")));
+
+ ctx->callbacks.stream_change_cb(ctx, txn, relation, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ int nrelations, Relation relations[],
+ ReorderBufferChange *change)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (!ctx->callbacks.stream_truncate_cb)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_truncate";
+ state.report_location = change->lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = change->lsn;
+
+ ctx->callbacks.stream_truncate_cb(ctx, txn, nrelations, relations, change);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn, bool transactional,
+ const char *prefix, Size message_size, const char *message)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* this callback is optional */
+ if (ctx->callbacks.stream_message_cb == NULL)
+ return;
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_message";
+ state.report_location = message_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
+ ctx->write_location = message_lsn;
+
+ /* do the actual work: call callback */
+ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix,
+ message_size, message);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_abort_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_abort";
+ state.report_location = abort_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = abort_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_abort_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_abort_cb callback.")));
+
+ ctx->callbacks.stream_abort_cb(ctx, txn, abort_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn,
+ XLogRecPtr apply_lsn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_commit";
+ state.report_location = apply_lsn;
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = apply_lsn;
+
+ /* in streaming mode, stream_abort_cb is required */
+ if (ctx->callbacks.stream_commit_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_commit_cb callback.")));
+
+ ctx->callbacks.stream_commit_cb(ctx, txn, apply_lsn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_start";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->first_lsn;
+
+ /* in streaming mode, stream_start_cb is required */
+ if (ctx->callbacks.stream_start_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_start_cb callback.")));
+
+ ctx->callbacks.stream_start_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
+static void
+stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
+{
+ LogicalDecodingContext *ctx = cache->private_data;
+ LogicalErrorCallbackState state;
+ ErrorContextCallback errcallback;
+
+ Assert(!ctx->fast_forward);
+
+ /* We're only supposed to call this when streaming is supported. */
+ Assert(ctx->streaming);
+
+ /* Push callback + info on the error context stack */
+ state.ctx = ctx;
+ state.callback_name = "stream_stop";
+ /* state.report_location = apply_lsn; */
+ errcallback.callback = output_plugin_error_callback;
+ errcallback.arg = (void *) &state;
+ errcallback.previous = error_context_stack;
+ error_context_stack = &errcallback;
+
+ /* set output state */
+ ctx->accept_writes = true;
+ ctx->write_xid = txn->xid;
+
+ /*
+ * report this change's lsn so replies from clients can give an up2date
+ * answer. This won't ever be enough (and shouldn't be!) to confirm
+ * receipt of this transaction, but it might allow another transaction's
+ * commit to be confirmed with one message.
+ */
+ ctx->write_location = txn->final_lsn;
+
+ /* in streaming mode, stream_stop_cb is required */
+ if (ctx->callbacks.stream_stop_cb == NULL)
+ ereport(ERROR,
+ (errmsg("Output plugin supports streaming, but has not registered "
+ "stream_stop_cb callback.")));
+
+ ctx->callbacks.stream_stop_cb(ctx, txn);
+
+ /* Pop the error context stack */
+ error_context_stack = errcallback.previous;
+}
+
/*
* Set the required catalog xmin horizon for historic snapshots in the current
* replication slot.
diff --git a/src/include/replication/logical.h b/src/include/replication/logical.h
index 3b7ca7f..f24e246 100644
--- a/src/include/replication/logical.h
+++ b/src/include/replication/logical.h
@@ -80,6 +80,11 @@ typedef struct LogicalDecodingContext
void *output_writer_private;
/*
+ * Does the output plugin support streaming, and is it enabled?
+ */
+ bool streaming;
+
+ /*
* State for writing output.
*/
bool accept_writes;
diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h
index 3dd9236..0d0a94a 100644
--- a/src/include/replication/output_plugin.h
+++ b/src/include/replication/output_plugin.h
@@ -100,6 +100,67 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct
typedef void (*LogicalDecodeShutdownCB) (struct LogicalDecodingContext *ctx);
/*
+ * Callback for streaming individual changes from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming truncates from in-progress transactions.
+ */
+typedef void (*LogicalDecodeStreamTruncateCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/*
+ * Callback for streaming generic logical decoding messages from in-progress
+ * transactions.
+ */
+typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix,
+ Size message_size,
+ const char *message);
+
+/*
+ * Called to discard changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamAbortCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/*
+ * Called to apply changes streamed to remote node from in-progress
+ * transaction.
+ */
+typedef void (*LogicalDecodeStreamCommitCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/*
+ * Called when starting to stream a block of changes from in-progress
+ * transaction (may be called repeatedly, if it's streamed in multiple
+ * chunks).
+ */
+typedef void (*LogicalDecodeStreamStartCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
+ * Called when stopping to stream a block of changes from in-progress
+ * transaction to a remote node (may be called repeatedly, if it's streamed
+ * in multiple chunks).
+ */
+typedef void (*LogicalDecodeStreamStopCB) (struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+/*
* Output plugin callbacks
*/
typedef struct OutputPluginCallbacks
@@ -112,6 +173,14 @@ typedef struct OutputPluginCallbacks
LogicalDecodeMessageCB message_cb;
LogicalDecodeFilterByOriginCB filter_by_origin_cb;
LogicalDecodeShutdownCB shutdown_cb;
+ /* streaming of changes */
+ LogicalDecodeStreamChangeCB stream_change_cb;
+ LogicalDecodeStreamTruncateCB stream_truncate_cb;
+ LogicalDecodeStreamMessageCB stream_message_cb;
+ LogicalDecodeStreamAbortCB stream_abort_cb;
+ LogicalDecodeStreamCommitCB stream_commit_cb;
+ LogicalDecodeStreamStartCB stream_start_cb;
+ LogicalDecodeStreamStopCB stream_stop_cb;
} OutputPluginCallbacks;
/* Functions in replication/logical/logical.c */
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index af35287..e102840 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -354,6 +354,52 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb,
const char *prefix, Size sz,
const char *message);
+/* stream change callback signature */
+typedef void (*ReorderBufferStreamChangeCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ Relation relation,
+ ReorderBufferChange *change);
+
+/* stream truncate callback signature */
+typedef void (*ReorderBufferStreamTruncateCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ int nrelations,
+ Relation relations[],
+ ReorderBufferChange *change);
+
+/* stream message callback signature */
+typedef void (*ReorderBufferStreamMessageCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr message_lsn,
+ bool transactional,
+ const char *prefix, Size sz,
+ const char *message);
+
+/* discard streamed transaction callback signature */
+typedef void (*ReorderBufferStreamAbortCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+/* commit streamed transaction callback signature */
+typedef void (*ReorderBufferStreamCommitCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+/* start streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStartCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
+/* stop streaming transaction callback signature */
+typedef void (*ReorderBufferStreamStopCB) (
+ ReorderBuffer *rb,
+ ReorderBufferTXN *txn);
+
struct ReorderBuffer
{
/*
@@ -393,6 +439,17 @@ struct ReorderBuffer
ReorderBufferMessageCB message;
/*
+ * Callbacks to be called when streaming a transaction.
+ */
+ ReorderBufferStreamStartCB stream_start;
+ ReorderBufferStreamStopCB stream_stop;
+ ReorderBufferStreamChangeCB stream_change;
+ ReorderBufferStreamTruncateCB stream_truncate;
+ ReorderBufferStreamMessageCB stream_message;
+ ReorderBufferStreamAbortCB stream_abort;
+ ReorderBufferStreamCommitCB stream_commit;
+
+ /*
* Pointer that will be passed untouched to the callbacks.
*/
void *private_data;
--
1.8.3.1
v16-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch (application/octet-stream, 12.3K) download
From 1ef5f6fbc9525a24f83e43cdeea9f46a13848cbd Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 9 Apr 2020 10:55:19 +0530
Subject: [PATCH v16 04/12] Gracefully handle concurrent aborts of uncommitted
transactions that are being decoded alongside.
When a transaction aborts, it's changes are considered unnecessary for
other transactions. That means the changes may be either cleaned up by
vacuum or removed from HOT chains (thus made inaccessible through
indexes), and there may be other such consequences.
When decoding committed transactions this is not an issue, and we
never decode transactions that abort before the decoding starts.
But for in-progress transactions, this may cause failures when the
output plugin consults catalogs (both system and user-defined).
We handle such failures by returning ERRCODE_TRANSACTION_ROLLBACK
sqlerrcode from system table scan APIs to the backend decoding a
specific uncommitted transaction. The decoding logic on the receipt
of such an sqlerrcode aborts the ongoing decoding and returns
gracefully.
---
doc/src/sgml/logicaldecoding.sgml | 5 ++-
src/backend/access/heap/heapam.c | 41 +++++++++++++++++++++++++
src/backend/access/index/genam.c | 38 +++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 8 ++---
src/backend/utils/time/snapmgr.c | 25 +++++++++++++--
src/include/utils/snapmgr.h | 4 ++-
6 files changed, 113 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml
index 65244b1..b59a6c8 100644
--- a/doc/src/sgml/logicaldecoding.sgml
+++ b/doc/src/sgml/logicaldecoding.sgml
@@ -432,7 +432,10 @@ typedef void (*LogicalOutputPluginInit) (struct OutputPluginCallbacks *cb);
ALTER TABLE user_catalog_table SET (user_catalog_table = true);
CREATE TABLE another_catalog_table(data text) WITH (user_catalog_table = true);
</programlisting>
- Any actions leading to transaction ID assignment are prohibited. That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the <literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment are prohibited. That, among others,
includes writing to tables, performing DDL changes, and
calling <literal>pg_current_xact_id()</literal>.
</para>
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 0d4ed60..84884a4 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1288,6 +1288,15 @@ heap_getnext(TableScanDesc sscan, ScanDirection direction)
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
errmsg_internal("only heap AM is supported")));
+ /*
+ * We don't expect direct calls to heap_getnext with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(scan->rs_base.rs_rd) ||
+ RelationIsUsedAsCatalogTable(scan->rs_base.rs_rd))))
+ elog(ERROR, "unexpected heap_getnext call during logical decoding");
+
/* Note: no locking manipulations needed */
if (scan->rs_base.rs_flags & SO_ALLOW_PAGEMODE)
@@ -1383,6 +1392,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
+ /*
* Fetch and pin the appropriate page of the relation.
*/
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
@@ -1497,6 +1514,14 @@ heap_hot_search_buffer(ItemPointer tid, Relation relation, Buffer buffer,
bool valid;
bool skip;
+ /*
+ * We don't expect direct calls to heap_hot_search_buffer with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_hot_search_buffer call during logical decoding");
+
/* If this is not the first call, previous call returned a (live!) tuple */
if (all_dead)
*all_dead = first_call;
@@ -1646,6 +1671,14 @@ heap_get_latest_tid(TableScanDesc sscan,
Assert(ItemPointerIsValid(tid));
/*
+ * We don't expect direct calls to heap_get_latest_tid with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_get_latest_tid call during logical decoding");
+
+ /*
* Loop to chase down t_ctid links. At top of loop, ctid is the tuple we
* need to examine, and *tid is the TID we will return if ctid turns out
* to be bogus.
@@ -5451,6 +5484,14 @@ heap_finish_speculative(Relation relation, ItemPointer tid)
ItemId lp = NULL;
HeapTupleHeader htup;
+ /*
+ * We don't expect direct calls to heap_finish_speculative with
+ * valid CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_finish_speculative call during logical decoding");
+
buffer = ReadBuffer(relation, ItemPointerGetBlockNumber(tid));
LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
page = (Page) BufferGetPage(buffer);
diff --git a/src/backend/access/index/genam.c b/src/backend/access/index/genam.c
index dfba5ae..97a1075 100644
--- a/src/backend/access/index/genam.c
+++ b/src/backend/access/index/genam.c
@@ -28,6 +28,7 @@
#include "lib/stringinfo.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/lsyscache.h"
@@ -433,6 +434,25 @@ systable_beginscan(Relation heapRelation,
}
/*
+ * HandleConcurrentAbort - Handle concurrent abort of the CheckXidAlive.
+ *
+ * If CheckXidAlive is valid, then we check if it aborted. If it did, we error
+ * out. We can't directly use TransactionIdDidAbort as after crash such
+ * transaction might not have been marked as aborted. See detailed comments at
+ * snapmgr.c where the variable is declared.
+ */
+static inline void
+HandleConcurrentAbort()
+{
+ if (TransactionIdIsValid(CheckXidAlive) &&
+ !TransactionIdIsInProgress(CheckXidAlive) &&
+ !TransactionIdDidCommit(CheckXidAlive))
+ ereport(ERROR,
+ (errcode(ERRCODE_TRANSACTION_ROLLBACK),
+ errmsg("transaction aborted during system catalog scan")));
+}
+
+/*
* systable_getnext --- get next tuple in a heap-or-index scan
*
* Returns NULL if no more tuples available.
@@ -481,6 +501,12 @@ systable_getnext(SysScanDesc sysscan)
}
}
+ /*
+ * Handle the concurrent abort while fetching the catalog tuple during
+ * logical streaming of a transaction.
+ */
+ HandleConcurrentAbort();
+
return htup;
}
@@ -517,6 +543,12 @@ systable_recheck_tuple(SysScanDesc sysscan, HeapTuple tup)
sysscan->slot,
freshsnap);
+ /*
+ * Handle the concurrent abort while fetching the catalog tuple during
+ * logical streaming of a transaction.
+ */
+ HandleConcurrentAbort();
+
return result;
}
@@ -643,6 +675,12 @@ systable_getnext_ordered(SysScanDesc sysscan, ScanDirection direction)
if (htup && sysscan->iscan->xs_recheck)
elog(ERROR, "system catalog scans with lossy index conditions are not implemented");
+ /*
+ * Handle the concurrent abort while fetching the catalog tuple during
+ * logical streaming of a transaction.
+ */
+ HandleConcurrentAbort();
+
return htup;
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 0d5bb73..2302875 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -696,7 +696,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
/* setup snapshot to allow catalog access */
- SetupHistoricSnapshot(snapshot_now, NULL);
+ SetupHistoricSnapshot(snapshot_now, NULL, xid);
PG_TRY();
{
rb->message(rb, txn, lsn, false, prefix, message_size, message);
@@ -1547,7 +1547,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1798,7 +1798,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1818,7 +1818,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
}
break;
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 1c063c5..93a0c04 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -154,6 +154,13 @@ static Snapshot CatalogSnapshot = NULL;
static Snapshot HistoricSnapshot = NULL;
/*
+ * An xid value pointing to a possibly ongoing (sub)transaction.
+ * Currently used in logical decoding. It's possible that such transactions
+ * can get aborted while the decoding is ongoing.
+ */
+TransactionId CheckXidAlive = InvalidTransactionId;
+
+/*
* These are updated by GetSnapshotData. We initialize them this way
* for the convenience of TransactionIdIsInProgress: even in bootstrap
* mode, we don't want it to say that BootstrapTransactionId is in progress.
@@ -2029,10 +2036,14 @@ MaintainOldSnapshotTimeMapping(TimestampTz whenTaken, TransactionId xmin)
* Setup a snapshot that replaces normal catalog snapshots that allows catalog
* access to behave just like it did at a certain point in the past.
*
+ * If a valid xid is passed in, we check if it is uncommitted and track it in
+ * CheckXidAlive. This is to re-check XID status while accessing catalog.
+ *
* Needed for logical decoding.
*/
void
-SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
+SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids,
+ TransactionId snapshot_xid)
{
Assert(historic_snapshot != NULL);
@@ -2041,8 +2052,17 @@ SetupHistoricSnapshot(Snapshot historic_snapshot, HTAB *tuplecids)
/* setup (cmin, cmax) lookup hash */
tuplecid_data = tuplecids;
-}
+ /*
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
+ */
+ if (TransactionIdIsValid(snapshot_xid) &&
+ !TransactionIdDidCommit(snapshot_xid))
+ CheckXidAlive = snapshot_xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
/*
* Make catalog snapshots behave normally again.
@@ -2052,6 +2072,7 @@ TeardownHistoricSnapshot(bool is_error)
{
HistoricSnapshot = NULL;
tuplecid_data = NULL;
+ CheckXidAlive = InvalidTransactionId;
}
bool
diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h
index b28d13c..12f737b 100644
--- a/src/include/utils/snapmgr.h
+++ b/src/include/utils/snapmgr.h
@@ -145,8 +145,10 @@ extern bool XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot);
/* Support for catalog timetravel for logical decoding */
struct HTAB;
+extern TransactionId CheckXidAlive;
extern struct HTAB *HistoricSnapshotGetTupleCids(void);
-extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids);
+extern void SetupHistoricSnapshot(Snapshot snapshot_now, struct HTAB *tuplecids,
+ TransactionId snapshot_xid);
extern void TeardownHistoricSnapshot(bool is_error);
extern bool HistoricSnapshotActive(void);
--
1.8.3.1
v16-0005-Implement-streaming-mode-in-ReorderBuffer.patch (application/octet-stream, 38.0K) download
From cdf1728d12f4440f0ab6ef3e4cc5ace6ae2645a7 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 10 Jan 2020 18:03:27 -0300
Subject: [PATCH v16 05/12] Implement streaming mode in ReorderBuffer
Instead of serializing the transaction to disk after reaching the
maximum number of changes in memory (4096 changes), we consume the
changes we have in memory and invoke new stream API methods. This
happens in ReorderBufferStreamTXN() using about the same logic as
in ReorderBufferCommit() logic.
We can do this incremental processing thanks to having assignments
(associating subxact with toplevel xacts) in WAL right away, and
thanks to logging the invalidation messages.
This adds a second iterator for the streaming case, without the
spill-to-disk functionality and only processing changes currently
in memory.
Theoretically, we could get rid of the k-way merge, and append the
changes to the toplevel xact directly (and remember the position
in the list in case the subxact gets aborted later).
It also adds ReorderBufferTXN pointer to two places:
* ReorderBufferChange, so that we know which xact it belongs to
* ReorderBufferTXN, pointing to toplevel xact (from subxact)
The output plugin can use this to decide which changes to discard
in case of stream_abort_cb (e.g. when a subxact gets discarded).
---
src/backend/access/heap/heapam_visibility.c | 38 +-
src/backend/replication/logical/reorderbuffer.c | 720 +++++++++++++++++++++---
src/include/replication/reorderbuffer.h | 36 ++
3 files changed, 702 insertions(+), 92 deletions(-)
diff --git a/src/backend/access/heap/heapam_visibility.c b/src/backend/access/heap/heapam_visibility.c
index dba1089..160b167 100644
--- a/src/backend/access/heap/heapam_visibility.c
+++ b/src/backend/access/heap/heapam_visibility.c
@@ -1571,8 +1571,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmin is
+ * definitely in the future, and we're not supposed to see the
+ * tuple yet.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
if (!resolved)
- elog(ERROR, "could not resolve cmin/cmax of catalog tuple");
+ return false;
Assert(cmin != InvalidCommandId);
@@ -1642,10 +1657,23 @@ HeapTupleSatisfiesHistoricMVCC(HeapTuple htup, Snapshot snapshot,
htup, buffer,
&cmin, &cmax);
- if (!resolved)
- elog(ERROR, "could not resolve combocid to cmax");
-
- Assert(cmax != InvalidCommandId);
+ /*
+ * If we haven't resolved the combocid to cmin/cmax, that means
+ * we have not decoded the combocid yet. That means the cmax is
+ * definitely in the future, and we're still supposed to see the
+ * tuple.
+ *
+ * XXX This only applies to decoding of in-progress transactions.
+ * In regular logical decoding we only execute this code at commit
+ * time, at which point we should have seen all relevant combocids.
+ * So we should error out in this case.
+ *
+ * XXX For the streaming case, we can track the largest combocid
+ * assigned, and error out based on this (when unable to resolve
+ * combocid below that observed maximum value).
+ */
+ if (!resolved || cmax == InvalidCommandId)
+ return true;
if (cmax >= snapshot->curcid)
return true; /* deleted after scan started */
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 2302875..1868ab2 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -236,6 +236,7 @@ static Size ReorderBufferRestoreChanges(ReorderBuffer *rb, ReorderBufferTXN *txn
static void ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
char *change);
static void ReorderBufferRestoreCleanup(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
static void ReorderBufferCleanupSerializedTXNs(const char *slotname);
static void ReorderBufferSerializedPath(char *path, ReplicationSlot *slot,
TransactionId xid, XLogSegNo segno);
@@ -244,6 +245,15 @@ static void ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap);
static Snapshot ReorderBufferCopySnap(ReorderBuffer *rb, Snapshot orig_snap,
ReorderBufferTXN *txn, CommandId cid);
+/*
+ * ---------------------------------------
+ * Streaming support functions
+ * ---------------------------------------
+ */
+static inline bool ReorderBufferCanStream(ReorderBuffer *rb);
+static void ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
/* ---------------------------------------
* toast reassembly support
* ---------------------------------------
@@ -371,6 +381,9 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ /* InvalidCommandId is not zero, so set it explicitly */
+ txn->command_id = InvalidCommandId;
+
return txn;
}
@@ -773,6 +786,38 @@ AssertTXNLsnOrder(ReorderBuffer *rb)
}
/*
+ * AssertChangeLsnOrder
+ *
+ * Check ordering of changes in the toplevel transaction.
+ */
+static void
+AssertChangeLsnOrder(ReorderBufferTXN *txn)
+{
+#ifdef USE_ASSERT_CHECKING
+ dlist_iter iter;
+ XLogRecPtr prev_lsn = txn->first_lsn;
+
+ dlist_foreach(iter, &txn->changes)
+ {
+ ReorderBufferChange *cur_change;
+
+ cur_change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ Assert(txn->first_lsn != InvalidXLogRecPtr);
+ Assert(cur_change->lsn != InvalidXLogRecPtr);
+ Assert(txn->first_lsn <= cur_change->lsn);
+
+ if (txn->end_lsn != InvalidXLogRecPtr)
+ Assert(cur_change->lsn <= txn->end_lsn);
+
+ Assert(prev_lsn <= cur_change->lsn);
+
+ prev_lsn = cur_change->lsn;
+ }
+#endif
+}
+
+/*
* ReorderBufferGetOldestTXN
* Return oldest transaction in reorderbuffer
*/
@@ -865,6 +910,9 @@ ReorderBufferAssignChild(ReorderBuffer *rb, TransactionId xid,
subtxn->toplevel_xid = xid;
Assert(subtxn->nsubtxns == 0);
+ /* set the reference to toplevel transaction */
+ subtxn->toptxn = txn;
+
/* add to subtransaction list */
dlist_push_tail(&txn->subtxns, &subtxn->node);
txn->nsubtxns++;
@@ -988,7 +1036,7 @@ ReorderBufferCommitChild(ReorderBuffer *rb, TransactionId xid,
*/
/*
- * Binary heap comparison function.
+ * Binary heap comparison function (regular non-streaming iterator).
*/
static int
ReorderBufferIterCompare(Datum a, Datum b, void *arg)
@@ -1024,6 +1072,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*iter_state = NULL;
+ /* Check ordering of changes in the toplevel transaction. */
+ AssertChangeLsnOrder(txn);
+
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
@@ -1038,6 +1089,9 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
+
if (cur_txn->nentries > 0)
nr_txns++;
}
@@ -1316,6 +1370,15 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Cleanup the snapshot for the last streamed run.
+ */
+ if (txn->snapshot_now != NULL)
+ {
+ Assert(rbtxn_is_streamed(txn));
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
* Remove TXN from its containing list.
*
* Note: if txn is known as subxact, we are deleting the TXN from its
@@ -1341,8 +1404,93 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
}
/*
+ * Discard changes from a transaction (and subtransactions), after streaming
+ * them. Keep the remaining info - transactions, tuplecids and snapshots.
+ */
+static void
+ReorderBufferTruncateTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ dlist_mutable_iter iter;
+
+ /* cleanup subtransactions & their changes */
+ dlist_foreach_modify(iter, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /*
+ * Subtransactions are always associated to the toplevel TXN, even if
+ * they originally were happening inside another subtxn, so we won't
+ * ever recurse more than one level deep here.
+ */
+ Assert(rbtxn_is_known_subxact(subtxn));
+ Assert(subtxn->nsubtxns == 0);
+
+ ReorderBufferTruncateTXN(rb, subtxn);
+ }
+
+ /* cleanup changes in the toplevel txn */
+ dlist_foreach_modify(iter, &txn->changes)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ /* remove the change from it's containing list */
+ dlist_delete(&change->node);
+
+ ReorderBufferReturnChange(rb, change);
+ }
+
+ /*
+ * Mark the transaction as streamed.
+ *
+ * The toplevel transaction, identified by (toptxn==NULL), is marked
+ * as streamed always, even if it does not contain any changes (that
+ * is, when all the changes are in subtransactions).
+ *
+ * For subtransactions, we only mark them as streamed when there are
+ * any changes in them.
+ *
+ * We do it this way because of aborts - we don't want to send aborts
+ * for XIDs the downstream is not aware of. And of course, it always
+ * knows about the toplevel xact (we send the XID in all messages),
+ * but we never stream XIDs of empty subxacts.
+ */
+ if ((!txn->toptxn) || (txn->nentries_mem != 0))
+ txn->txn_flags |= RBTXN_IS_STREAMED;
+
+ /*
+ * Destroy the (relfilenode, ctid) hashtable, so that we don't leak
+ * any memory. We could also keep the hash table and update it with
+ * new ctid values, but this seems simpler and good enough for now.
+ */
+ if (txn->tuplecid_hash != NULL)
+ {
+ hash_destroy(txn->tuplecid_hash);
+ txn->tuplecid_hash = NULL;
+ }
+
+ /* also reset the number of entries in the transaction */
+ txn->nentries_mem = 0;
+ txn->nentries = 0;
+}
+
+/*
* Build a hash with a (relfilenode, ctid) -> (cmin, cmax) mapping for use by
- * HeapTupleSatisfiesHistoricMVCC.
+ * tqual.c's HeapTupleSatisfiesHistoricMVCC.
+ *
+ * We do build the hash table even if there are no CIDs. That's
+ * because when streaming in-progress transactions we may run into
+ * tuples with the CID before actually decoding them. Think e.g. about
+ * INSERT followed by TRUNCATE, where the TRUNCATE may not be decoded
+ * yet when applying the INSERT. So we build a hash table so that
+ * ResolveCminCmaxDuringDecoding does not segfault in this case.
+ *
+ * XXX We might limit this behavior to streaming mode, and just bail
+ * out when decoding transaction at commit time (at which point it's
+ * guaranteed to see all CIDs).
*/
static void
ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
@@ -1350,9 +1498,6 @@ ReorderBufferBuildTupleCidHash(ReorderBuffer *rb, ReorderBufferTXN *txn)
dlist_iter iter;
HASHCTL hash_ctl;
- if (!rbtxn_has_catalog_changes(txn) || dlist_is_empty(&txn->tuplecids))
- return;
-
memset(&hash_ctl, 0, sizeof(hash_ctl));
hash_ctl.keysize = sizeof(ReorderBufferTupleCidKey);
@@ -1491,63 +1636,76 @@ ReorderBufferFreeSnap(ReorderBuffer *rb, Snapshot snap)
}
/*
- * Perform the replay of a transaction and its non-aborted subtransactions.
- *
- * Subtransactions previously have to be processed by
- * ReorderBufferCommitChild(), even if previously assigned to the toplevel
- * transaction with ReorderBufferAssignChild.
- *
- * We currently can only decode a transaction's contents when its commit
- * record is read because that's the only place where we know about cache
- * invalidations. Thus, once a toplevel commit is read, we iterate over the top
- * and subtransactions (using a k-way merge) and replay the changes in lsn
- * order.
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
*/
-void
-ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
- TimestampTz commit_time,
- RepOriginId origin_id, XLogRecPtr origin_lsn)
+static void
+ReorderBufferStreamCommit(ReorderBuffer *rb, ReorderBufferTXN *txn)
{
- ReorderBufferTXN *txn;
- volatile Snapshot snapshot_now;
- volatile CommandId command_id = FirstCommandId;
- bool using_subtxn;
- ReorderBufferIterTXNState *volatile iterstate = NULL;
+ /* we should only call this for previously streamed transactions */
+ Assert(rbtxn_is_streamed(txn));
- txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
- false);
+ ReorderBufferStreamTXN(rb, txn);
- /* unknown transaction, nothing to replay */
- if (txn == NULL)
- return;
+ rb->stream_commit(rb, txn, txn->final_lsn);
- txn->final_lsn = commit_lsn;
- txn->end_lsn = end_lsn;
- txn->commit_time = commit_time;
- txn->origin_id = origin_id;
- txn->origin_lsn = origin_lsn;
+ ReorderBufferCleanupTXN(rb, txn);
+}
+/*
+ * Set xid for concurrent abort check
+ *
+ * While streaming an in-progress transaction there is a possibility that the
+ * (sub)transaction might get aborted concurrently. In such case if the
+ * (sub)transaction has catalog update then we might decode the tuple using
+ * wrong catalog version. So for detecting the concurrent abort we set
+ * CheckXidAlive to the current (sub)transaction's xid for which this change
+ * belongs to. And, during catalog scan we can check the status of the xid and
+ * if it is aborted we will report a specific error that we can ignore. We
+ * might have already streamed some of the changes for the aborted
+ * (sub)transaction, but that is fine because when we decode the abort we will
+ * stream abort message to truncate the changes in the subscriber.
+ */
+static inline void
+SetupCheckXidLive(TransactionId xid)
+{
/*
- * If this transaction has no snapshot, it didn't make any changes to the
- * database, so there's nothing to decode. Note that
- * ReorderBufferCommitChild will have transferred any snapshots from
- * subtransactions if there were any.
+ * setup CheckXidAlive if it's not committed yet. We don't check
+ * if the xid aborted. That will happen during catalog access.
*/
- if (txn->base_snapshot == NULL)
- {
- Assert(txn->ninvalidations == 0);
- ReorderBufferCleanupTXN(rb, txn);
- return;
- }
+ if (!TransactionIdDidCommit(xid))
+ CheckXidAlive = xid;
+ else
+ CheckXidAlive = InvalidTransactionId;
+}
- snapshot_now = txn->base_snapshot;
+/*
+ * Helper function for ReorderBufferCommit and ReorderBufferStreamTXN
+ *
+ * Send data of a transaction (and its subtransactions) to the
+ * output plugin. If streaming is true then data will be sent using stream API.
+ */
+static void
+ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn,
+ volatile Snapshot snapshot_now,
+ volatile CommandId command_id,
+ bool streaming)
+{
+ bool using_subtxn;
+ MemoryContext ccxt = CurrentMemoryContext;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+ volatile XLogRecPtr prev_lsn = InvalidXLogRecPtr;
+ ReorderBufferChange *volatile specinsert = NULL;
- /* build data to be able to lookup the CommandIds of catalog tuples */
+ /*
+ * build data to be able to lookup the CommandIds of catalog tuples
+ */
ReorderBufferBuildTupleCidHash(rb, txn);
/* setup the initial snapshot */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, txn->xid);
/*
* Decoding needs access to syscaches et al., which in turn use
@@ -1564,14 +1722,17 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
PG_TRY();
{
ReorderBufferChange *change;
- ReorderBufferChange *specinsert = NULL;
if (using_subtxn)
- BeginInternalSubTransaction("replay");
+ BeginInternalSubTransaction("stream");
else
StartTransactionCommand();
- rb->begin(rb, txn);
+ /* start streaming this chunk of transaction */
+ if (streaming)
+ rb->stream_start(rb, txn);
+ else
+ rb->begin(rb, txn);
ReorderBufferIterTXNInit(rb, txn, &iterstate);
while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
@@ -1579,6 +1740,19 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
Relation relation = NULL;
Oid reloid;
+ /*
+ * Enforce correct ordering of changes, merged from multiple
+ * subtransactions. The changes may have the same LSN due to
+ * MULTI_INSERT xlog records.
+ */
+ Assert(prev_lsn == InvalidXLogRecPtr || prev_lsn <= change->lsn);
+
+ prev_lsn = change->lsn;
+
+ /* Set the xid for concurrent abort check. */
+ if (streaming)
+ SetupCheckXidLive(change->txn->xid);
+
switch (change->action)
{
case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
@@ -1588,8 +1762,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* use as a normal record. It'll be cleaned up at the end
* of INSERT processing.
*/
- if (specinsert == NULL)
- elog(ERROR, "invalid ordering of speculative insertion changes");
Assert(specinsert->data.tp.oldtuple == NULL);
change = specinsert;
change->action = REORDER_BUFFER_CHANGE_INSERT;
@@ -1655,7 +1827,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (!IsToastRelation(relation))
{
ReorderBufferToastReplace(rb, txn, relation, change);
- rb->apply_change(rb, txn, relation, change);
+ if (streaming)
+ {
+ rb->stream_change(rb, txn, relation, change);
+
+ /* Remember that we have sent some data for this xid. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_change(rb, txn, relation, change);
/*
* Only clear reassembled toast chunks if we're sure
@@ -1676,8 +1856,6 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* freed/reused while restoring spooled data from
* disk.
*/
- Assert(change->data.tp.newtuple != NULL);
-
dlist_delete(&change->node);
ReorderBufferToastAppendChunk(rb, txn, relation,
change);
@@ -1695,7 +1873,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
specinsert = NULL;
}
- if (relation != NULL)
+ if (RelationIsValid(relation))
{
RelationClose(relation);
relation = NULL;
@@ -1753,7 +1931,15 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
relations[nrelations++] = relation;
}
- rb->apply_truncate(rb, txn, nrelations, relations, change);
+ if (streaming)
+ {
+ rb->stream_truncate(rb, txn, nrelations, relations, change);
+
+ /* Remember that we have sent some data. */
+ change->txn->any_data_sent = true;
+ }
+ else
+ rb->apply_truncate(rb, txn, nrelations, relations, change);
for (i = 0; i < nrelations; i++)
RelationClose(relations[i]);
@@ -1762,10 +1948,16 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
}
case REORDER_BUFFER_CHANGE_MESSAGE:
- rb->message(rb, txn, change->lsn, true,
- change->data.msg.prefix,
- change->data.msg.message_size,
- change->data.msg.message);
+ if (streaming)
+ rb->stream_message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
+ else
+ rb->message(rb, txn, change->lsn, true,
+ change->data.msg.prefix,
+ change->data.msg.message_size,
+ change->data.msg.message);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -1796,9 +1988,9 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now = change->data.snapshot;
}
-
/* and continue with the new one */
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
break;
case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
@@ -1818,7 +2010,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
snapshot_now->curcid = command_id;
TeardownHistoricSnapshot(false);
- SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash, xid);
+ SetupHistoricSnapshot(snapshot_now, txn->tuplecid_hash,
+ txn->xid);
}
break;
@@ -1858,14 +2051,46 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
ReorderBufferIterTXNFinish(rb, iterstate);
iterstate = NULL;
- /* call commit callback */
- rb->commit(rb, txn, commit_lsn);
+ /*
+ * Done with current changes, call stream_stop callback for streaming
+ * transaction, commit callback otherwise.
+ */
+ if (streaming)
+ {
+ /*
+ * Set the last of the stream as the final lsn before calling
+ * stream stop.
+ */
+ if (!XLogRecPtrIsInvalid(prev_lsn))
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ }
+ else
+ rb->commit(rb, txn, commit_lsn);
/* this is just a sanity check against bad output plugin behaviour */
if (GetCurrentTransactionIdIfAny() != InvalidTransactionId)
elog(ERROR, "output plugin used XID %u",
GetCurrentTransactionId());
+ /*
+ * Remember the command ID and snapshot if transaction is streaming
+ * otherwise free the snapshot if we have copied it.
+ */
+ if (streaming)
+ {
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ }
+ else if (snapshot_now->copied)
+ ReorderBufferFreeSnap(rb, snapshot_now);
+
/* cleanup */
TeardownHistoricSnapshot(false);
@@ -1884,14 +2109,22 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ /*
+ * If we are streaming the in-progress transaction then discard the
+ * changes that we just streamed, and mark the transactions as streamed
+ * (if they contained changes). Otherwise, remove all the changes and
+ * deallocate the ReorderBufferTXN.
+ */
+ if (streaming)
+ ReorderBufferTruncateTXN(rb, txn);
+ else
+ ReorderBufferCleanupTXN(rb, txn);
}
PG_CATCH();
{
+ MemoryContext ecxt = MemoryContextSwitchTo(ccxt);
+ ErrorData *errdata = CopyErrorData();
+
/* TODO: Encapsulate cleanup from the PG_TRY and PG_CATCH blocks */
if (iterstate)
ReorderBufferIterTXNFinish(rb, iterstate);
@@ -1911,18 +2144,131 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
- if (snapshot_now->copied)
- ReorderBufferFreeSnap(rb, snapshot_now);
-
- /* remove potential on-disk data, and deallocate */
- ReorderBufferCleanupTXN(rb, txn);
+ if (streaming)
+ {
+ /* Discard the changes that we just streamed. */
+ ReorderBufferTruncateTXN(rb, txn);
- PG_RE_THROW();
+ /* Re-throw only if it's not an abort. */
+ if (errdata->sqlerrcode != ERRCODE_TRANSACTION_ROLLBACK)
+ {
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
+ else
+ {
+ FlushErrorState();
+ FreeErrorData(errdata);
+ errdata = NULL;
+
+ /* remember the command ID and snapshot for the streaming run */
+ txn->command_id = command_id;
+
+ /* Avoid copying if it's already copied. */
+ if (snapshot_now->copied)
+ txn->snapshot_now = snapshot_now;
+ else
+ txn->snapshot_now = ReorderBufferCopySnap(rb, snapshot_now,
+ txn, command_id);
+ /*
+ * Set the last last of the stream as the final lsn before
+ * calling stream stop.
+ */
+ txn->final_lsn = prev_lsn;
+ rb->stream_stop(rb, txn);
+ ReorderBufferToastReset(rb, txn);
+ if (specinsert != NULL)
+ {
+ ReorderBufferReturnChange(rb, specinsert);
+ specinsert = NULL;
+ }
+ }
+ }
+ else
+ {
+ ReorderBufferCleanupTXN(rb, txn);
+ MemoryContextSwitchTo(ecxt);
+ PG_RE_THROW();
+ }
}
PG_END_TRY();
}
/*
+ * Perform the replay of a transaction and its non-aborted subtransactions.
+ *
+ * Subtransactions previously have to be processed by
+ * ReorderBufferCommitChild(), even if previously assigned to the toplevel
+ * transaction with ReorderBufferAssignChild.
+ *
+ * We currently can only decode a transaction's contents when its commit
+ * record is read because that's the only place where we know about cache
+ * invalidations. Thus, once a toplevel commit is read, we iterate over the top
+ * and subtransactions (using a k-way merge) and replay the changes in lsn
+ * order.
+ */
+void
+ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr commit_lsn, XLogRecPtr end_lsn,
+ TimestampTz commit_time,
+ RepOriginId origin_id, XLogRecPtr origin_lsn)
+{
+ ReorderBufferTXN *txn;
+ Snapshot snapshot_now;
+ CommandId command_id = FirstCommandId;
+
+ txn = ReorderBufferTXNByXid(rb, xid, false, NULL, InvalidXLogRecPtr,
+ false);
+
+ /* unknown transaction, nothing to replay */
+ if (txn == NULL)
+ return;
+
+ txn->final_lsn = commit_lsn;
+ txn->end_lsn = end_lsn;
+ txn->commit_time = commit_time;
+ txn->origin_id = origin_id;
+ txn->origin_lsn = origin_lsn;
+
+ /*
+ * If the transaction was (partially) streamed, we need to commit it in a
+ * 'streamed' way. That is, we first stream the remaining part of the
+ * transaction, and then invoke stream_commit message.
+ *
+ * XXX Called after everything (origin ID and LSN, ...) is stored in the
+ * transaction, so we don't pass that directly.
+ *
+ * XXX Somewhat hackish redirection, perhaps needs to be refactored?
+ */
+ if (rbtxn_is_streamed(txn))
+ {
+ ReorderBufferStreamCommit(rb, txn);
+ return;
+ }
+
+ /*
+ * If this transaction has no snapshot, it didn't make any changes to the
+ * database, so there's nothing to decode. Note that
+ * ReorderBufferCommitChild will have transferred any snapshots from
+ * subtransactions if there were any.
+ */
+ if (txn->base_snapshot == NULL)
+ {
+ Assert(txn->ninvalidations == 0);
+ ReorderBufferCleanupTXN(rb, txn);
+ return;
+ }
+
+ snapshot_now = txn->base_snapshot;
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, commit_lsn, snapshot_now,
+ command_id, false);
+}
+
+/*
* Abort a transaction that possibly has previous changes. Needs to be first
* called for subtransactions and then for the toplevel xid.
*
@@ -1946,6 +2292,13 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * If the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2015,6 +2368,13 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
if (txn == NULL)
return;
+ /*
+ * If the (sub)transaction was streamed, notify the remote node
+ * about the abort only if we have sent any data for this transaction.
+ */
+ if (rbtxn_is_streamed(txn) && txn->any_data_sent)
+ rb->stream_abort(rb, txn, lsn);
+
/* cosmetic... */
txn->final_lsn = lsn;
@@ -2150,8 +2510,17 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Update the memory accounting info. We track memory used by the whole
- * reorder buffer and the transaction containing the change.
+ * Update memory counters to account for the new or removed change.
+ *
+ * We update two counters - in the reorder buffer, and in the transaction
+ * containing the change. The reorder buffer counter allows us to quickly
+ * decide if we reached the memory limit, the transaction counter allows
+ * us to quickly pick the largest transaction for eviction.
+ *
+ * When streaming is enabled, we need to update the toplevel transaction
+ * counters instead - we don't really care about subtransactions as we
+ * can't stream them individually anyway, and we only pick toplevel
+ * transactions for eviction. So only toplevel transactions matter.
*/
static void
ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
@@ -2159,6 +2528,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
bool addition)
{
Size sz;
+ ReorderBufferTXN *txn;
Assert(change->txn);
@@ -2170,19 +2540,28 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID)
return;
+ txn = change->txn;
+
+ /* if subxact, and streaming supported, use the toplevel instead */
+ if (txn->toptxn && ReorderBufferCanStream(rb))
+ txn = txn->toptxn;
+
sz = ReorderBufferChangeSize(change);
if (addition)
{
- change->txn->size += sz;
+ txn->size += sz;
rb->size += sz;
}
else
{
- Assert((rb->size >= sz) && (change->txn->size >= sz));
- change->txn->size -= sz;
+ Assert((rb->size >= sz) && (txn->size >= sz));
+ txn->size -= sz;
rb->size -= sz;
}
+
+ Assert(txn->size <= rb->size);
+ Assert((txn->size >= 0) && (rb->size >= 0));
}
/*
@@ -2211,6 +2590,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
change->lsn = lsn;
change->txn = txn;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID;
+ change->txn = txn;
dlist_push_tail(&txn->tuplecids, &change->node);
txn->ntuplecids++;
@@ -2295,6 +2675,13 @@ ReorderBufferXidSetCatalogChanges(ReorderBuffer *rb, TransactionId xid,
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
txn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
+
+ /*
+ * TOCHECK: Mark toplevel transaction as having catalog changes too
+ * if one of its children has.
+ */
+ if (txn->toptxn != NULL)
+ txn->toptxn->txn_flags |= RBTXN_HAS_CATALOG_CHANGES;
}
/*
@@ -2399,6 +2786,38 @@ ReorderBufferLargestTXN(ReorderBuffer *rb)
}
/*
+ * Find the largest toplevel transaction to evict (by streaming).
+ *
+ * This can be seen as an optimized version of ReorderBufferLargestTXN, which
+ * should give us the same transaction (because we don't update memory account
+ * for subtransaction with streaming, so it's always 0). But we can simply
+ * iterate over the limited number of toplevel transactions.
+ */
+static ReorderBufferTXN *
+ReorderBufferLargestTopTXN(ReorderBuffer *rb)
+{
+ dlist_iter iter;
+ ReorderBufferTXN *largest = NULL;
+
+ dlist_foreach(iter, &rb->toplevel_by_lsn)
+ {
+ ReorderBufferTXN *txn;
+
+ txn = dlist_container(ReorderBufferTXN, node, iter.cur);
+
+ /* if the current transaction is larger, remember it */
+ if ((!largest) || (txn->size > largest->size))
+ largest = txn;
+ }
+
+ Assert(largest);
+ Assert(largest->size > 0);
+ Assert(largest->size <= rb->size);
+
+ return largest;
+}
+
+/*
* Check whether the logical_decoding_work_mem limit was reached, and if yes
* pick the transaction to evict and spill the changes to disk.
*
@@ -2418,15 +2837,46 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
/*
* Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTXN(rb);
+ if (ReorderBufferCanStream(rb))
+ {
+ /*
+ * Pick the largest toplevel transaction and evict it from memory by
+ * streaming the already decoded part.
+ */
+ txn = ReorderBufferLargestTopTXN(rb);
+
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- ReorderBufferSerializeTXN(rb, txn);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
+
+ ReorderBufferSerializeTXN(rb, txn);
+ }
/*
* After eviction, the transaction should have no entries in memory, and
* should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
*/
Assert(txn->size == 0);
Assert(txn->nentries_mem == 0);
@@ -2746,6 +3196,102 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(ondisk->change.action == change->action);
}
+static inline bool
+ReorderBufferCanStream(ReorderBuffer *rb)
+{
+ LogicalDecodingContext *ctx = rb->private_data;
+
+ return ctx->streaming;
+}
+
+/*
+ * Send data of a large transaction (and its subtransactions) to the
+ * output plugin, but using the stream API.
+ *
+ * XXX Do we need to check if the transaction has some changes to stream
+ * (maybe it got streamed right before the commit, which attempts to
+ * stream it again before the commit)?
+ */
+static void
+ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /* We can never reach here for a sub transaction. */
+ Assert(txn->toptxn == NULL);
+
+ /*
+ * XXX Not sure if we can make any assumptions about base snapshot here,
+ * similarly to what ReorderBufferCommit() does. That relies on
+ * base_snapshot getting transferred from subxact in
+ * ReorderBufferCommitChild(), but that was not yet called as the
+ * transaction is in-progress.
+ *
+ * So just walk the subxacts and use the same logic here. But we only need
+ * to do that once, when the transaction is streamed for the first time.
+ * After that we need to reuse the snapshot from the previous run.
+ */
+ if (txn->snapshot_now == NULL)
+ {
+ dlist_iter subxact_i;
+
+ /* make sure this transaction is streamed for the first time */
+ Assert(!rbtxn_is_streamed(txn));
+
+ /* at the beginning we should have invalid command ID */
+ Assert(txn->command_id == InvalidCommandId);
+
+ dlist_foreach(subxact_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *subtxn;
+
+ subtxn = dlist_container(ReorderBufferTXN, node, subxact_i.cur);
+ ReorderBufferTransferSnapToParent(txn, subtxn);
+ }
+
+ command_id = FirstCommandId;
+ snapshot_now = ReorderBufferCopySnap(rb, txn->base_snapshot,
+ txn, command_id);
+ }
+ else
+ {
+ /* the transaction must have been already streamed */
+ Assert(rbtxn_is_streamed(txn));
+
+ /*
+ * Nah, we already have snapshot from the previous streaming run. We
+ * assume new subxacts can't move the LSN backwards, and so can't beat
+ * the LSN condition in the previous branch (so no need to walk
+ * through subxacts again). In fact, we must not do that as we may be
+ * using snapshot half-way through the subxact.
+ */
+ command_id = txn->command_id;
+
+ /*
+ * We can not use txn->snapshot_now directly because after the last
+ * streaming run, we might have got some new sub-transactions. So we
+ * need to add them to the snapshot.
+ */
+ snapshot_now = ReorderBufferCopySnap(rb, txn->snapshot_now,
+ txn, command_id);
+
+ /* Free the previously copied snapshot. */
+ Assert(txn->snapshot_now->copied);
+ ReorderBufferFreeSnap(rb, txn->snapshot_now);
+ }
+
+ /*
+ * Access the main routine to decode the changes and send to output plugin.
+ */
+ ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
+ command_id, true);
+
+ Assert(dlist_is_empty(&txn->changes));
+ Assert(txn->nentries == 0);
+ Assert(txn->nentries_mem == 0);
+}
+
/*
* Size of a change in memory.
*/
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index e102840..6d65986 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -171,6 +171,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_STREAMED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -190,6 +191,24 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/*
+ * Has this transaction been streamed to downstream?
+ *
+ * (It's not possible to deduce this from nentries and nentries_mem for
+ * various reasons. For example, all changes may be in subtransactions in
+ * which case we'd have nentries==0 for the toplevel one, which would say
+ * nothing about the streaming. So we maintain this flag, but only for the
+ * toplevel transaction.)
+ *
+ * Note: We never do both stream and serialize a transaction (we only spill
+ * to disk when streaming is not supported by the plugin), so only one of
+ * those two flags may be set at any given time.
+ */
+#define rbtxn_is_streamed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_IS_STREAMED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -225,6 +244,16 @@ typedef struct ReorderBufferTXN
XLogRecPtr final_lsn;
/*
+ * Have we sent any changes for this transaction in output plugin?
+ */
+ bool any_data_sent;
+
+ /*
+ * Toplevel transaction for this subxact (NULL for top-level).
+ */
+ struct ReorderBufferTXN *toptxn;
+
+ /*
* LSN pointing to the end of the commit record + 1.
*/
XLogRecPtr end_lsn;
@@ -255,6 +284,13 @@ typedef struct ReorderBufferTXN
dlist_node base_snapshot_node; /* link in txns_by_base_snapshot_lsn */
/*
+ * Snapshot/CID from the previous streaming run. Only valid for already
+ * streamed transactions (NULL/InvalidCommandId otherwise).
+ */
+ Snapshot snapshot_now;
+ CommandId command_id;
+
+ /*
* How many ReorderBufferChange's do we have in this txn.
*
* Changes in subtransactions are *not* included but tracked separately.
--
1.8.3.1
v16-0009-Add-TAP-test-for-streaming-vs.-DDL.patch (application/octet-stream, 4.4K) download
From e71bd7c12db18f2caf2885e9279374d79983ef55 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tv@fuzzy.cz>
Date: Thu, 26 Sep 2019 19:15:35 +0200
Subject: [PATCH v16 09/12] Add TAP test for streaming vs. DDL
---
src/test/subscription/t/014_stream_through_ddl.pl | 98 +++++++++++++++++++++++
1 file changed, 98 insertions(+)
create mode 100644 src/test/subscription/t/014_stream_through_ddl.pl
diff --git a/src/test/subscription/t/014_stream_through_ddl.pl b/src/test/subscription/t/014_stream_through_ddl.pl
new file mode 100644
index 0000000..b8d78b1
--- /dev/null
+++ b/src/test/subscription/t/014_stream_through_ddl.pl
@@ -0,0 +1,98 @@
+# Test streaming of large transaction with DDL, subtransactions and rollbacks.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d text, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming=true)"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 1000) s(i);
+SAVEPOINT s2;
+ALTER TABLE test_tab ADD COLUMN c INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(1001, 2000) s(i);
+SAVEPOINT s3;
+ALTER TABLE test_tab ADD COLUMN d text;
+SAVEPOINT s4;
+SAVEPOINT s5;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text) FROM generate_series(2001, 3000) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(3001, 4000) s(i);
+SAVEPOINT s10;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(4001, 5000) s(i);
+ALTER TABLE test_tab ADD COLUMN d text;
+ROLLBACK TO SAVEPOINT s10;
+RELEASE SAVEPOINT s10;
+SAVEPOINT s10;
+INSERT INTO test_tab SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(5001, 6000) s(i);
+SAVEPOINT s6;
+ALTER TABLE test_tab DROP d;
+INSERT INTO test_tab SELECT i, md5(i::text), i, i FROM generate_series(6001, 7000) s(i);
+SAVEPOINT s7;
+ALTER TABLE test_tab ADD COLUMN d text;
+INSERT INTO test_tab (a, b, c, d, e) SELECT i, md5(i::text), i, md5(i::text), i FROM generate_series(7001, 8000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(a), count(b), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(7000|7000|7000|6000|4000|4000), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v16-0006-Add-support-for-streaming-to-built-in-replicatio.patch (application/octet-stream, 91.3K) download
From 1b8a05e44a991237ccd0ac109ecc1018cba9c946 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Thu, 16 Apr 2020 01:55:22 -0700
Subject: [PATCH v16 06/12] Add support for streaming to built-in replication
To add support for streaming of in-progress transactions into the
built-in transaction, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
---
doc/src/sgml/ref/alter_subscription.sgml | 4 +-
doc/src/sgml/ref/create_subscription.sgml | 12 +
src/backend/catalog/pg_subscription.c | 1 +
src/backend/commands/subscriptioncmds.c | 45 +-
src/backend/postmaster/pgstat.c | 12 +
.../libpqwalreceiver/libpqwalreceiver.c | 3 +
src/backend/replication/logical/launcher.c | 1 -
src/backend/replication/logical/logical.c | 4 +-
src/backend/replication/logical/proto.c | 140 ++-
src/backend/replication/logical/worker.c | 1035 ++++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 318 +++++-
src/backend/replication/slotfuncs.c | 6 +
src/backend/replication/walsender.c | 6 +
src/include/catalog/pg_subscription.h | 3 +
src/include/pgstat.h | 6 +-
src/include/replication/logicalproto.h | 42 +-
src/include/replication/walreceiver.h | 1 +
src/test/subscription/t/009_stream_simple.pl | 86 ++
src/test/subscription/t/010_stream_subxact.pl | 102 ++
src/test/subscription/t/011_stream_ddl.pl | 95 ++
.../subscription/t/012_stream_subxact_abort.pl | 82 ++
.../subscription/t/013_stream_subxact_ddl_abort.pl | 84 ++
22 files changed, 2045 insertions(+), 43 deletions(-)
create mode 100644 src/test/subscription/t/009_stream_simple.pl
create mode 100644 src/test/subscription/t/010_stream_subxact.pl
create mode 100644 src/test/subscription/t/011_stream_ddl.pl
create mode 100644 src/test/subscription/t/012_stream_subxact_abort.pl
create mode 100644 src/test/subscription/t/013_stream_subxact_ddl_abort.pl
diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml
index 1b8bead..95b7c24 100644
--- a/doc/src/sgml/ref/alter_subscription.sgml
+++ b/doc/src/sgml/ref/alter_subscription.sgml
@@ -164,8 +164,8 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO <
<para>
This clause alters parameters originally set by
<xref linkend="sql-createsubscription"/>. See there for more
- information. The allowed options are <literal>slot_name</literal> and
- <literal>synchronous_commit</literal>
+ information. The allowed options are <literal>slot_name</literal>,
+ <literal>synchronous_commit</literal> and <literal>streaming</literal>.
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index 1a90c24..3349cc4 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -206,6 +206,18 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>streaming</literal> (<type>boolean</type>)</term>
+ <listitem>
+ <para>
+ Specifies whether streaming of in-progress transactions should
+ be enabled for this subscription. By default, all transactions
+ are fully decoded on the publisher, and only then sent to the
+ subscriber as a whole.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
</para>
</listitem>
diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c
index cb15731..f28482f 100644
--- a/src/backend/catalog/pg_subscription.c
+++ b/src/backend/catalog/pg_subscription.c
@@ -65,6 +65,7 @@ GetSubscription(Oid subid, bool missing_ok)
sub->name = pstrdup(NameStr(subform->subname));
sub->owner = subform->subowner;
sub->enabled = subform->subenabled;
+ sub->stream = subform->substream;
/* Get conninfo */
datum = SysCacheGetAttr(SUBSCRIPTIONOID,
diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c
index 7f15667..65b6b76 100644
--- a/src/backend/commands/subscriptioncmds.c
+++ b/src/backend/commands/subscriptioncmds.c
@@ -59,7 +59,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
bool *enabled, bool *create_slot,
bool *slot_name_given, char **slot_name,
bool *copy_data, char **synchronous_commit,
- bool *refresh)
+ bool *refresh, bool *streaming,
+ bool *streaming_given)
{
ListCell *lc;
bool connect_given = false;
@@ -90,6 +91,8 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
*synchronous_commit = NULL;
if (refresh)
*refresh = true;
+ if (streaming)
+ *streaming_given = false;
/* Parse options */
foreach(lc, options)
@@ -175,6 +178,16 @@ parse_subscription_options(List *options, bool *connect, bool *enabled_given,
refresh_given = true;
*refresh = defGetBoolean(defel);
}
+ else if (strcmp(defel->defname, "streaming") == 0 && streaming)
+ {
+ if (*streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+
+ *streaming_given = true;
+ *streaming = defGetBoolean(defel);
+ }
else
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
@@ -318,6 +331,8 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
bool enabled_given;
bool enabled;
bool copy_data;
+ bool streaming;
+ bool streaming_given;
char *synchronous_commit;
char *conninfo;
char *slotname;
@@ -334,7 +349,7 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
parse_subscription_options(stmt->options, &connect, &enabled_given,
&enabled, &create_slot, &slotname_given,
&slotname, ©_data, &synchronous_commit,
- NULL);
+ NULL, &streaming, &streaming_given);
/*
* Since creating a replication slot is not transactional, rolling back
@@ -412,6 +427,13 @@ CreateSubscription(CreateSubscriptionStmt *stmt, bool isTopLevel)
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(publications);
+ if (streaming_given)
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ else
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(false);
+
tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
/* Insert tuple into catalog. */
@@ -669,10 +691,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
char *slotname;
bool slotname_given;
char *synchronous_commit;
+ bool streaming;
+ bool streaming_given;
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, &slotname_given, &slotname,
- NULL, &synchronous_commit, NULL);
+ NULL, &synchronous_commit, NULL,
+ &streaming, &streaming_given);
if (slotname_given)
{
@@ -697,6 +722,13 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
replaces[Anum_pg_subscription_subsynccommit - 1] = true;
}
+ if (streaming_given)
+ {
+ values[Anum_pg_subscription_substream - 1] =
+ BoolGetDatum(streaming);
+ replaces[Anum_pg_subscription_substream - 1] = true;
+ }
+
update_tuple = true;
break;
}
@@ -708,7 +740,8 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL,
&enabled_given, &enabled, NULL,
- NULL, NULL, NULL, NULL, NULL);
+ NULL, NULL, NULL, NULL, NULL,
+ NULL, NULL);
Assert(enabled_given);
if (!sub->slotname && enabled)
@@ -746,7 +779,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, &refresh);
+ NULL, &refresh, NULL, NULL);
values[Anum_pg_subscription_subpublications - 1] =
publicationListToArray(stmt->publication);
@@ -783,7 +816,7 @@ AlterSubscription(AlterSubscriptionStmt *stmt)
parse_subscription_options(stmt->options, NULL, NULL, NULL,
NULL, NULL, NULL, ©_data,
- NULL, NULL);
+ NULL, NULL, NULL, NULL);
AlterSubscription_refresh(sub, copy_data);
diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c
index 50eea2e..4ef4fd4 100644
--- a/src/backend/postmaster/pgstat.c
+++ b/src/backend/postmaster/pgstat.c
@@ -4133,6 +4133,18 @@ pgstat_get_wait_io(WaitEventIO w)
case WAIT_EVENT_WAL_WRITE:
event_name = "WALWrite";
break;
+ case WAIT_EVENT_LOGICAL_CHANGES_READ:
+ event_name = "ReorderLogicalChangesRead";
+ break;
+ case WAIT_EVENT_LOGICAL_CHANGES_WRITE:
+ event_name = "ReorderLogicalChangesWrite";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_READ:
+ event_name = "ReorderLogicalSubxactRead";
+ break;
+ case WAIT_EVENT_LOGICAL_SUBXACT_WRITE:
+ event_name = "ReorderLogicalSubxactWrite";
+ break;
/* no default case, so that compiler will warn */
}
diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index e4fd1f9..5257ab0 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -408,6 +408,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn,
appendStringInfo(&cmd, "proto_version '%u'",
options->proto.logical.proto_version);
+ if (options->proto.logical.streaming)
+ appendStringInfo(&cmd, ", streaming 'on'");
+
pubnames = options->proto.logical.publication_names;
pubnames_str = stringlist_to_identifierstr(conn->streamConn, pubnames);
if (!pubnames_str)
diff --git a/src/backend/replication/logical/launcher.c b/src/backend/replication/logical/launcher.c
index aec885e..8156a42 100644
--- a/src/backend/replication/logical/launcher.c
+++ b/src/backend/replication/logical/launcher.c
@@ -14,7 +14,6 @@
*
*-------------------------------------------------------------------------
*/
-
#include "postgres.h"
#include "access/heapam.h"
diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c
index 497d8a9..dfc681d 100644
--- a/src/backend/replication/logical/logical.c
+++ b/src/backend/replication/logical/logical.c
@@ -1148,7 +1148,7 @@ stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_start";
- /* state.report_location = apply_lsn; */
+ state.report_location = InvalidXLogRecPtr;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
@@ -1193,7 +1193,7 @@ stream_stop_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn)
/* Push callback + info on the error context stack */
state.ctx = ctx;
state.callback_name = "stream_stop";
- /* state.report_location = apply_lsn; */
+ state.report_location = txn->final_lsn;
errcallback.callback = output_plugin_error_callback;
errcallback.arg = (void *) &state;
errcallback.previous = error_context_stack;
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 3c6d0cd..5242ac0 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -139,10 +139,15 @@ logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn)
* Write INSERT to the output stream.
*/
void
-logicalrep_write_insert(StringInfo out, Relation rel, HeapTuple newtuple)
+logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple)
{
pq_sendbyte(out, 'I'); /* action INSERT */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -178,8 +183,8 @@ logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup)
* Write UPDATE to the output stream.
*/
void
-logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
- HeapTuple newtuple)
+logicalrep_write_update(StringInfo out, TransactionId xid, Relation rel,
+ HeapTuple oldtuple, HeapTuple newtuple)
{
pq_sendbyte(out, 'U'); /* action UPDATE */
@@ -187,6 +192,10 @@ logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_INDEX);
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -248,7 +257,8 @@ logicalrep_read_update(StringInfo in, bool *has_oldtuple,
* Write DELETE to the output stream.
*/
void
-logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
+logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple)
{
Assert(rel->rd_rel->relreplident == REPLICA_IDENTITY_DEFAULT ||
rel->rd_rel->relreplident == REPLICA_IDENTITY_FULL ||
@@ -256,6 +266,10 @@ logicalrep_write_delete(StringInfo out, Relation rel, HeapTuple oldtuple)
pq_sendbyte(out, 'D'); /* action DELETE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -296,6 +310,7 @@ logicalrep_read_delete(StringInfo in, LogicalRepTupleData *oldtup)
*/
void
logicalrep_write_truncate(StringInfo out,
+ TransactionId xid,
int nrelids,
Oid relids[],
bool cascade, bool restart_seqs)
@@ -305,6 +320,10 @@ logicalrep_write_truncate(StringInfo out,
pq_sendbyte(out, 'T'); /* action TRUNCATE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
pq_sendint32(out, nrelids);
/* encode and send truncate flags */
@@ -347,12 +366,16 @@ logicalrep_read_truncate(StringInfo in,
* Write relation description to the output stream.
*/
void
-logicalrep_write_rel(StringInfo out, Relation rel)
+logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel)
{
char *relname;
pq_sendbyte(out, 'R'); /* sending RELATION */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
/* use Oid as relation identifier */
pq_sendint32(out, RelationGetRelid(rel));
@@ -397,7 +420,7 @@ logicalrep_read_rel(StringInfo in)
* This function will always write base type info.
*/
void
-logicalrep_write_typ(StringInfo out, Oid typoid)
+logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid)
{
Oid basetypoid = getBaseType(typoid);
HeapTuple tup;
@@ -405,6 +428,10 @@ logicalrep_write_typ(StringInfo out, Oid typoid)
pq_sendbyte(out, 'Y'); /* sending TYPE */
+ /* transaction ID (if not valid, we're not streaming) */
+ if (TransactionIdIsValid(xid))
+ pq_sendint32(out, xid);
+
tup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(basetypoid));
if (!HeapTupleIsValid(tup))
elog(ERROR, "cache lookup failed for type %u", basetypoid);
@@ -685,3 +712,104 @@ logicalrep_read_namespace(StringInfo in)
return nspname;
}
+
+void
+logicalrep_write_stream_start(StringInfo out,
+ TransactionId xid, bool first_segment)
+{
+ pq_sendbyte(out, 'S'); /* action STREAM START */
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+
+ /* 1 if this is the first streaming segment for this xid */
+ pq_sendbyte(out, first_segment ? 1 : 0);
+}
+
+TransactionId
+logicalrep_read_stream_start(StringInfo in, bool *first_segment)
+{
+ TransactionId xid;
+
+ Assert(first_segment);
+
+ xid = pq_getmsgint(in, 4);
+ *first_segment = (pq_getmsgbyte(in) == 1);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_stop(StringInfo out)
+{
+ pq_sendbyte(out, 'E'); /* action STREAM END */
+}
+
+void
+logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ uint8 flags = 0;
+
+ pq_sendbyte(out, 'c'); /* action STREAM COMMIT */
+
+ Assert(TransactionIdIsValid(txn->xid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, txn->xid);
+
+ /* send the flags field (unused for now) */
+ pq_sendbyte(out, flags);
+
+ /* send fields */
+ pq_sendint64(out, commit_lsn);
+ pq_sendint64(out, txn->end_lsn);
+ pq_sendint64(out, txn->commit_time);
+}
+
+TransactionId
+logicalrep_read_stream_commit(StringInfo in, LogicalRepCommitData *commit_data)
+{
+ TransactionId xid;
+ uint8 flags;
+
+ xid = pq_getmsgint(in, 4);
+
+ /* read flags (unused for now) */
+ flags = pq_getmsgbyte(in);
+
+ if (flags != 0)
+ elog(ERROR, "unrecognized flags %u in commit message", flags);
+
+ /* read fields */
+ commit_data->commit_lsn = pq_getmsgint64(in);
+ commit_data->end_lsn = pq_getmsgint64(in);
+ commit_data->committime = pq_getmsgint64(in);
+
+ return xid;
+}
+
+void
+logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid)
+{
+ pq_sendbyte(out, 'A'); /* action STREAM ABORT */
+
+ Assert(TransactionIdIsValid(xid) && TransactionIdIsValid(subxid));
+
+ /* transaction ID (we're starting to stream, so must be valid) */
+ pq_sendint32(out, xid);
+ pq_sendint32(out, subxid);
+}
+
+void
+logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid)
+{
+ Assert(xid && subxid);
+
+ *xid = pq_getmsgint(in, 4);
+ *subxid = pq_getmsgint(in, 4);
+}
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a752a12..05e7954 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -18,11 +18,32 @@
* This module includes server facing code and shares libpqwalreceiver
* module with walreceiver for providing the libpq specific functionality.
*
+ *
+ * STREAMED TRANSACTIONS
+ * ---------------------
+ *
+ * Streamed transactions (large transactions exceeding a memory limit on the
+ * upstream) are not applied immediately, but instead the data is written
+ * to files and then applied at once when the final commit arrives.
+ *
+ * Unlike the regular (non-streamed) case, handling streamed transactions has
+ * to handle aborts of both the toplevel transaction and subtransactions. This
+ * is achieved by tracking offsets for subtransactions, which is then used
+ * to truncate the file with serialized changes.
+ *
+ * The files are placed in tmp file directory by default, and the filenames
+ * include both the XID of the toplevel transaction and OID of the subscription.
+ * This is necessary so that different workers processing a remote transaction
+ * with the same XID don't interfere.
+ *
*-------------------------------------------------------------------------
*/
#include "postgres.h"
+#include <sys/stat.h>
+#include <unistd.h>
+
#include "access/table.h"
#include "access/tableam.h"
#include "access/xact.h"
@@ -33,6 +54,7 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "catalog/pg_tablespace.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -64,6 +86,7 @@
#include "replication/worker_internal.h"
#include "rewrite/rewriteHandler.h"
#include "storage/bufmgr.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lmgr.h"
#include "storage/proc.h"
@@ -71,6 +94,7 @@
#include "tcop/tcopprot.h"
#include "utils/builtins.h"
#include "utils/catcache.h"
+#include "utils/dynahash.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -110,12 +134,57 @@ bool MySubscriptionValid = false;
bool in_remote_transaction = false;
static XLogRecPtr remote_final_lsn = InvalidXLogRecPtr;
+/* fields valid only when processing streamed transaction */
+bool in_streamed_transaction = false;
+
+static TransactionId stream_xid = InvalidTransactionId;
+static int stream_fd = -1;
+
+typedef struct SubXactInfo
+{
+ TransactionId xid; /* XID of the subxact */
+ off_t offset; /* offset in the file */
+} SubXactInfo;
+
+static uint32 nsubxacts = 0;
+static uint32 nsubxacts_max = 0;
+static SubXactInfo * subxacts = NULL;
+static TransactionId subxact_last = InvalidTransactionId;
+
+static void subxact_filename(char *path, Oid subid, TransactionId xid);
+static void changes_filename(char *path, Oid subid, TransactionId xid);
+
+/*
+ * Information about subtransactions of a given toplevel transaction.
+ */
+static void subxact_info_write(Oid subid, TransactionId xid);
+static void subxact_info_read(Oid subid, TransactionId xid);
+static void subxact_info_add(TransactionId xid);
+
+/*
+ * Serialize and deserialize changes for a toplevel transaction.
+ */
+static void stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok);
+static void stream_open_file(Oid subid, TransactionId xid, bool first);
+static void stream_write_change(char action, StringInfo s);
+static void stream_close_file(void);
+
+/*
+ * Array of serialized XIDs.
+ */
+static int nxids = 0;
+static int maxnxids = 0;
+static TransactionId *xids = NULL;
+
static void send_feedback(XLogRecPtr recvpos, bool force, bool requestReply);
static void store_flush_position(XLogRecPtr remote_lsn);
static void maybe_reread_subscription(void);
+/* prototype needed because of stream_commit */
+static void apply_dispatch(StringInfo s);
+
static void apply_handle_insert_internal(ResultRelInfo *relinfo,
EState *estate, TupleTableSlot *remoteslot);
static void apply_handle_update_internal(ResultRelInfo *relinfo,
@@ -187,6 +256,42 @@ ensure_transaction(void)
return true;
}
+/*
+ * Handle streamed transactions.
+ *
+ * If in streaming mode (receiving a block of streamed transaction), we
+ * simply redirect it to a file for the proper toplevel transaction.
+ *
+ * Returns true for streamed transactions, false otherwise (regular mode).
+ */
+static bool
+handle_streamed_transaction(const char action, StringInfo s)
+{
+ TransactionId xid;
+
+ /* not in streaming mode */
+ if (!in_streamed_transaction)
+ return false;
+
+ Assert(stream_fd != -1);
+ Assert(TransactionIdIsValid(stream_xid));
+
+ /*
+ * We should have received XID of the subxact as the first part of the
+ * message, so extract it.
+ */
+ xid = pq_getmsgint(s, 4);
+
+ Assert(TransactionIdIsValid(xid));
+
+ /* Add the new subxact to the array (unless already there). */
+ subxact_info_add(xid);
+
+ /* write the change to the current file */
+ stream_write_change(action, s);
+
+ return true;
+}
/*
* Executor state preparation for evaluation of constraint expressions,
@@ -553,6 +658,321 @@ apply_handle_origin(StringInfo s)
}
/*
+ * Handle STREAM START message.
+ */
+static void
+apply_handle_stream_start(StringInfo s)
+{
+ bool first_segment;
+
+ Assert(!in_streamed_transaction);
+
+ /* notify handle methods we're processing a remote transaction */
+ in_streamed_transaction = true;
+
+ /* extract XID of the top-level transaction */
+ stream_xid = logicalrep_read_stream_start(s, &first_segment);
+
+ /* open the spool file for this transaction */
+ stream_open_file(MyLogicalRepWorker->subid, stream_xid, first_segment);
+
+ /*
+ * if this is not the first segment, open existing file
+ *
+ * XXX Note that the cleanup is performed by stream_open_file.
+ */
+ if (!first_segment)
+ subxact_info_read(MyLogicalRepWorker->subid, stream_xid);
+
+ pgstat_report_activity(STATE_RUNNING, NULL);
+}
+
+/*
+ * Handle STREAM STOP message.
+ */
+static void
+apply_handle_stream_stop(StringInfo s)
+{
+ Assert(in_streamed_transaction);
+
+ /*
+ * Close the file with serialized changes, and serialize information about
+ * subxacts for the toplevel transaction.
+ */
+ subxact_info_write(MyLogicalRepWorker->subid, stream_xid);
+ stream_close_file();
+
+ in_streamed_transaction = false;
+
+ pgstat_report_activity(STATE_IDLE, NULL);
+}
+
+/*
+ * Handle STREAM abort message.
+ */
+static void
+apply_handle_stream_abort(StringInfo s)
+{
+ TransactionId xid;
+ TransactionId subxid;
+
+ Assert(!in_streamed_transaction);
+
+ logicalrep_read_stream_abort(s, &xid, &subxid);
+
+ /*
+ * If the two XIDs are the same, it's in fact abort of toplevel xact, so
+ * just delete the files with serialized info.
+ */
+ if (xid == subxid)
+ {
+ char path[MAXPGPATH];
+
+ /*
+ * XXX Maybe this should be an error instead? Can we receive abort for
+ * a toplevel transaction we haven't received?
+ */
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ subxact_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (unlink(path) < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ return;
+ }
+ else
+ {
+ /*
+ * OK, so it's a subxact. We need to read the subxact file for the
+ * toplevel transaction, determine the offset tracked for the subxact,
+ * and truncate the file with changes. We also remove the subxacts
+ * with higher offsets (or rather higher XIDs).
+ *
+ * We intentionally scan the array from the tail, because we're likely
+ * aborting a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ *
+ * XXX Or perhaps we can rely on the aborts to arrive in the reverse
+ * order, i.e. from the inner-most subxact (when nested)? In which
+ * case we could simply check the last element.
+ */
+
+ int64 i;
+ int64 subidx;
+ bool found = false;
+ char path[MAXPGPATH];
+
+ subidx = -1;
+ subxact_info_read(MyLogicalRepWorker->subid, xid);
+
+ /* XXX optimize the search by bsearch on sorted data */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ if (subxacts[i - 1].xid == subxid)
+ {
+ subidx = (i - 1);
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /* OK, truncate the file at the right offset. */
+ Assert((subidx >= 0) && (subidx < nsubxacts));
+
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ if (truncate(path, subxacts[subidx].offset))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not truncate file \"%s\": %m", path)));
+
+ /* discard the subxacts added later */
+ nsubxacts = subidx;
+
+ /* write the updated subxact list */
+ subxact_info_write(MyLogicalRepWorker->subid, xid);
+ }
+}
+
+/*
+ * Handle STREAM COMMIT message.
+ */
+static void
+apply_handle_stream_commit(StringInfo s)
+{
+ int fd;
+ TransactionId xid;
+ StringInfoData s2;
+ int nchanges;
+
+ char path[MAXPGPATH];
+ char *buffer = NULL;
+ LogicalRepCommitData commit_data;
+
+ MemoryContext oldcxt;
+
+ Assert(!in_streamed_transaction);
+
+ xid = logicalrep_read_stream_commit(s, &commit_data);
+
+ elog(DEBUG1, "received commit for streamed transaction %u", xid);
+
+ /* open the spool file for the committed transaction */
+ changes_filename(path, MyLogicalRepWorker->subid, xid);
+
+ elog(DEBUG1, "replaying changes from file '%s'", path);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ }
+
+ /* XXX Should this be allocated in another memory context? */
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ buffer = palloc(8192);
+ initStringInfo(&s2);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ ensure_transaction();
+
+ /*
+ * Make sure the handle apply_dispatch methods are aware we're in a remote
+ * transaction.
+ */
+ in_remote_transaction = true;
+ pgstat_report_activity(STATE_RUNNING, NULL);
+
+ /*
+ * Read the entries one by one and pass them through the same logic as in
+ * apply_dispatch.
+ */
+ nchanges = 0;
+ while (true)
+ {
+ int nbytes;
+ int len;
+
+ /* read length of the on-disk record */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ nbytes = read(fd, &len, sizeof(len));
+ pgstat_report_wait_end();
+
+ /* have we reached end of the file? */
+ if (nbytes == 0)
+ break;
+
+ /* do we have a correct length? */
+ if (nbytes != sizeof(len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+
+ Assert(len > 0);
+
+ /* make sure we have sufficiently large buffer */
+ buffer = repalloc(buffer, len);
+
+ /* and finally read the data into the buffer */
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_READ);
+ if (read(fd, buffer, len) != len)
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file: %m")));
+ return;
+ }
+ pgstat_report_wait_end();
+
+ /* copy the buffer to the stringinfo and call apply_dispatch */
+ resetStringInfo(&s2);
+ appendBinaryStringInfo(&s2, buffer, len);
+
+ /* Ensure we are reading the data into our memory context. */
+ oldcxt = MemoryContextSwitchTo(ApplyMessageContext);
+
+ apply_dispatch(&s2);
+
+ MemoryContextReset(ApplyMessageContext);
+
+ MemoryContextSwitchTo(oldcxt);
+
+ nchanges++;
+
+ if (nchanges % 1000 == 0)
+ elog(DEBUG1, "replayed %d changes from file '%s'",
+ nchanges, path);
+
+ /*
+ * send feedback to upstream
+ *
+ * XXX Probably should send a valid LSN. But which one?
+ */
+ send_feedback(InvalidXLogRecPtr, false, false);
+ }
+
+ if (CloseTransientFile(fd) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not close file \"%s\": %m", path)));
+
+ /*
+ * Update origin state so we can restart streaming from correct
+ * position in case of crash.
+ */
+ replorigin_session_origin_lsn = commit_data.end_lsn;
+ replorigin_session_origin_timestamp = commit_data.committime;
+
+ CommitTransactionCommand();
+ pgstat_report_stat(false);
+
+ store_flush_position(commit_data.end_lsn);
+
+ elog(DEBUG1, "replayed %d (all) changes from file '%s'",
+ nchanges, path);
+
+ in_remote_transaction = false;
+ pgstat_report_activity(STATE_IDLE, NULL);
+
+ /* unlink the files with serialized changes and subxact info */
+ stream_cleanup_files(MyLogicalRepWorker->subid, xid, false);
+
+ pfree(buffer);
+ pfree(s2.data);
+}
+
+/*
* Handle RELATION message.
*
* Note we don't do validation against local schema here. The validation
@@ -565,6 +985,9 @@ apply_handle_relation(StringInfo s)
{
LogicalRepRelation *rel;
+ if (handle_streamed_transaction('R', s))
+ return;
+
rel = logicalrep_read_rel(s);
logicalrep_relmap_update(rel);
}
@@ -580,6 +1003,9 @@ apply_handle_type(StringInfo s)
{
LogicalRepTyp typ;
+ if (handle_streamed_transaction('Y', s))
+ return;
+
logicalrep_read_typ(s, &typ);
logicalrep_typmap_update(&typ);
}
@@ -616,6 +1042,9 @@ apply_handle_insert(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('I', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_insert(s, &newtup);
@@ -731,6 +1160,9 @@ apply_handle_update(StringInfo s)
RangeTblEntry *target_rte;
MemoryContext oldctx;
+ if (handle_streamed_transaction('U', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_update(s, &has_oldtup, &oldtup,
@@ -873,6 +1305,9 @@ apply_handle_delete(StringInfo s)
TupleTableSlot *remoteslot;
MemoryContext oldctx;
+ if (handle_streamed_transaction('D', s))
+ return;
+
ensure_transaction();
relid = logicalrep_read_delete(s, &oldtup);
@@ -1243,6 +1678,9 @@ apply_handle_truncate(StringInfo s)
List *relids_logged = NIL;
ListCell *lc;
+ if (handle_streamed_transaction('T', s))
+ return;
+
ensure_transaction();
remote_relids = logicalrep_read_truncate(s, &cascade, &restart_seqs);
@@ -1381,6 +1819,22 @@ apply_dispatch(StringInfo s)
case 'O':
apply_handle_origin(s);
break;
+ /* STREAM START */
+ case 'S':
+ apply_handle_stream_start(s);
+ break;
+ /* STREAM END */
+ case 'E':
+ apply_handle_stream_stop(s);
+ break;
+ /* STREAM ABORT */
+ case 'A':
+ apply_handle_stream_abort(s);
+ break;
+ /* STREAM COMMIT */
+ case 'c':
+ apply_handle_stream_commit(s);
+ break;
default:
ereport(ERROR,
(errcode(ERRCODE_PROTOCOL_VIOLATION),
@@ -1478,6 +1932,22 @@ UpdateWorkerStats(XLogRecPtr last_lsn, TimestampTz send_time, bool reply)
}
/*
+ * Cleanup function.
+ *
+ * Called on logical replication worker exit.
+ */
+static void
+worker_onexit(int code, Datum arg)
+{
+ int i;
+
+ elog(LOG, "cleanup files for %d transactions", nxids);
+
+ for (i = nxids-1; i >= 0; i--)
+ stream_cleanup_files(MyLogicalRepWorker->subid, xids[i], true);
+}
+
+/*
* Apply main loop.
*/
static void
@@ -1493,6 +1963,9 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
"ApplyMessageContext",
ALLOCSET_DEFAULT_SIZES);
+ /* do cleanup on worker exit (e.g. after DROP SUBSCRIPTION) */
+ before_shmem_exit(worker_onexit, (Datum) 0);
+
/* mark as idle, before starting to loop */
pgstat_report_activity(STATE_IDLE, NULL);
@@ -1941,6 +2414,567 @@ subscription_change_cb(Datum arg, int cacheid, uint32 hashvalue)
MySubscriptionValid = false;
}
+/*
+ * subxact_info_write
+ * Store information about subxacts for a toplevel transaction.
+ *
+ * For each subxact we store offset of it's first change in the main file.
+ * The file is always over-written as a whole, and we also include CRC32C
+ * checksum of the information.
+ *
+ * XXX We should only store subxacts that were not aborted yet.
+ *
+ * XXX Maybe we should only include the checksum when the cluster is
+ * initialized with checksums?
+ */
+static void
+subxact_info_write(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ Size len;
+
+ Assert(TransactionIdIsValid(xid));
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_CREAT | O_TRUNC | O_WRONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* compute the checksum */
+ INIT_CRC32C(checksum);
+ COMP_CRC32C(checksum, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum, (char *) subxacts, len);
+ FIN_CRC32C(checksum);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);
+
+ if (write(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if (write(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ if ((len > 0) && (write(fd, subxacts, len) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not write to file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /*
+ * We don't need to fsync or anything, as we'll recreate the files after a
+ * crash from scratch. So just close the file.
+ */
+ if (CloseTransientFile(fd) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not close file \"%s\": %m", path)));
+
+ /*
+ * But we free the memory allocated for subxact info. There might be one
+ * exceptional transaction with many subxacts, and we don't want to keep
+ * the memory allocated forewer.
+ *
+ */
+ if (subxacts)
+ pfree(subxacts);
+
+ subxacts = NULL;
+ subxact_last = InvalidTransactionId;
+ nsubxacts = 0;
+ nsubxacts_max = 0;
+}
+
+/*
+ * subxact_info_read
+ * Restore information about subxacts of a streamed transaction.
+ *
+ * Read information about subxacts into the global variables, and while
+ * reading the information verify the checksum.
+ *
+ * XXX Add calls to pgstat_report_wait_start/pgstat_report_wait_end.
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_read(Oid subid, TransactionId xid)
+{
+ int fd;
+ char path[MAXPGPATH];
+ uint32 checksum;
+ uint32 checksum_new;
+ Size len;
+ MemoryContext oldctx;
+
+ Assert(TransactionIdIsValid(xid));
+ Assert(!subxacts);
+ Assert(nsubxacts == 0);
+ Assert(nsubxacts_max == 0);
+
+ subxact_filename(path, subid, xid);
+
+ fd = OpenTransientFile(path, O_RDONLY | PG_BINARY);
+ if (fd < 0)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ /* read the checksum */
+ if (read(fd, &checksum, sizeof(checksum)) != sizeof(checksum))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ /* read number of subxact items */
+ if (read(fd, &nsubxacts, sizeof(nsubxacts)) != sizeof(nsubxacts))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ len = sizeof(SubXactInfo) * nsubxacts;
+
+ /* we keep the maximum as a power of 2 */
+ nsubxacts_max = 1 << my_log2(nsubxacts);
+
+ /* subxacts are long-lived */
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_READ);
+
+ if ((len > 0) && ((read(fd, subxacts, len)) != len))
+ {
+ int save_errno = errno;
+
+ CloseTransientFile(fd);
+ errno = save_errno;
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not read file \"%s\": %m",
+ path)));
+ return;
+ }
+
+ pgstat_report_wait_end();
+
+ /* recompute the checksum */
+ INIT_CRC32C(checksum_new);
+ COMP_CRC32C(checksum_new, (char *) &nsubxacts, sizeof(nsubxacts));
+ COMP_CRC32C(checksum_new, (char *) subxacts, len);
+ FIN_CRC32C(checksum_new);
+
+ if (checksum_new != checksum)
+ ereport(ERROR,
+ (errmsg("checksum failure when reading subxacts")));
+
+ if (CloseTransientFile(fd) != 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not close file \"%s\": %m", path)));
+}
+
+/*
+ * subxact_info_add
+ * Add information about a subxact (offset in the main file).
+ *
+ * XXX Do we need to allocate it in TopMemoryContext?
+ */
+static void
+subxact_info_add(TransactionId xid)
+{
+ int64 i;
+
+ /*
+ * If the XID matches the toplevel transaction, we don't want to add it.
+ */
+ if (stream_xid == xid)
+ return;
+
+ /*
+ * In most cases we're checking the same subxact as we've already seen in
+ * the last call, so make ure just ignore it (this change comes later).
+ */
+ if (subxact_last == xid)
+ return;
+
+ /* OK, remember we're processing this XID. */
+ subxact_last = xid;
+
+ /*
+ * Check if the transaction is already present in the array of subxact. We
+ * intentionally scan the array from the tail, because we're likely adding
+ * a change for the most recent subtransactions.
+ *
+ * XXX Can we rely on the subxact XIDs arriving in sorted order? That
+ * would allow us to use binary search here.
+ */
+ for (i = nsubxacts; i > 0; i--)
+ {
+ /* found, so we're done */
+ if (subxacts[i - 1].xid == xid)
+ return;
+ }
+
+ /* This is a new subxact, so we need to add it to the array. */
+
+ if (nsubxacts == 0)
+ {
+ MemoryContext oldctx;
+
+ nsubxacts_max = 128;
+ oldctx = MemoryContextSwitchTo(TopMemoryContext);
+ subxacts = palloc(nsubxacts_max * sizeof(SubXactInfo));
+ MemoryContextSwitchTo(oldctx);
+ }
+ else if (nsubxacts == nsubxacts_max)
+ {
+ nsubxacts_max *= 2;
+ subxacts = repalloc(subxacts, nsubxacts_max * sizeof(SubXactInfo));
+ }
+
+ subxacts[nsubxacts].xid = xid;
+ subxacts[nsubxacts].offset = lseek(stream_fd, 0, SEEK_END);
+
+ nsubxacts++;
+}
+
+/* format filename for file containing the info about subxacts */
+static void
+subxact_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ */
+ if ((MakePGDirectory(tempdirpath) < 0) && errno != EEXIST)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create directory \"%s\": %m",
+ tempdirpath)));
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.subxacts",
+ tempdirpath, subid, xid);
+}
+
+/* format filename for file containing serialized changes */
+static void
+changes_filename(char *path, Oid subid, TransactionId xid)
+{
+ char tempdirpath[MAXPGPATH];
+
+ TempTablespacePath(tempdirpath, DEFAULTTABLESPACE_OID);
+
+ /*
+ * We might need to create the tablespace's tempfile directory, if no
+ * one has yet done so.
+ */
+ if ((MakePGDirectory(tempdirpath) < 0) && errno != EEXIST)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not create directory \"%s\": %m",
+ tempdirpath)));
+
+ snprintf(path, MAXPGPATH, "%s/logical-%u-%u.changes",
+ tempdirpath, subid, xid);
+}
+
+/*
+ * stream_cleanup_files
+ * Cleanup files for a subscription / toplevel transaction.
+ *
+ * Remove files with serialized changes and subxact info for a particular
+ * toplevel transaction. Each subscription has a separate set of files.
+ *
+ * Note: The files may not exists, so handle ENOENT as non-error.
+ *
+ * missing_ok - don't report error for missing file is the flag is passed true.
+ */
+static void
+stream_cleanup_files(Oid subid, TransactionId xid, bool missing_ok)
+{
+ int i;
+ char path[MAXPGPATH];
+ bool found = false;
+
+ subxact_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ changes_filename(path, subid, xid);
+
+ if ((unlink(path) < 0) && (errno != ENOENT) && !missing_ok)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+
+ /*
+ * Cleanup the XID from the array - find the XID in the array and
+ * remove it by shifting all the remaining elements. The array is
+ * bound to be fairly small (maximum number of in-progress xacts,
+ * so max_connections + max_prepared_transactions) so simply loop
+ * through the array and find index of the XID. Then move the rest
+ * of the array by one element to the left.
+ *
+ * Notice we also call this from stream_open_file for first segment
+ * of each transaction, to deal with possible left-overs after a
+ * crash, so it's entirely possible not to find the XID in the
+ * array here. In that case we don't remove anything.
+ *
+ * XXX Perhaps it'd be better to handle this automatically after a
+ * restart, instead of doing it over and over for each transaction.
+ */
+ for (i = 0; i < nxids; i++)
+ {
+ if (xids[i] == xid)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ return;
+
+ /*
+ * Move the last entry from the array to the place. We don't keep
+ * the streamed transactions sorted or anything - we only expect
+ * a few of them in progress (max_connections + max_prepared_xacts)
+ * so linear search is just fine.
+ */
+ xids[i] = xids[nxids-1];
+ nxids--;
+}
+
+/*
+ * stream_open_file
+ * Open file we'll use to serialize changes for a toplevel transaction.
+ *
+ * Open a file for streamed changes from a toplevel transaction identified
+ * by stream_xid (global variable). If it's the first chunk of streamed
+ * changes for this transaction, perform cleanup by removing existing
+ * files after a possible previous crash.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_open_file(Oid subid, TransactionId xid, bool first_segment)
+{
+ char path[MAXPGPATH];
+ int flags;
+
+ Assert(in_streamed_transaction);
+ Assert(OidIsValid(subid));
+ Assert(TransactionIdIsValid(xid));
+ Assert(stream_fd == -1);
+
+ /*
+ * If this is the first segment for this transaction, try removing
+ * existing files (if there are any, possibly after a crash).
+ */
+ if (first_segment)
+ {
+ MemoryContext oldcxt;
+
+ /* XXX make sure there are no previous files for this transaction */
+ stream_cleanup_files(subid, xid, true);
+
+ oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * We need to remember the XIDs we spilled to files, so that we can
+ * remove them at worker exit (e.g. after DROP SUBSCRIPTION).
+ *
+ * The number of XIDs we may need to track is fairly small, because
+ * we can only stream toplevel xacts (so limited by max_connections
+ * and max_prepared_transactions), and we only stream the large ones.
+ * So we simply keep the XIDs in an unsorted array. If the number of
+ * xacts gets large for some reason (e.g. very high max_connections),
+ * a more elaborate approach might be better - e.g. sorted array, to
+ * speed-up the lookups.
+ */
+ if (nxids == maxnxids) /* array of XIDs is full */
+ {
+ if (!xids)
+ {
+ maxnxids = 64;
+ xids = palloc(maxnxids * sizeof(TransactionId));
+ }
+ else
+ {
+ maxnxids = 2 * maxnxids;
+ xids = repalloc(xids, maxnxids * sizeof(TransactionId));
+ }
+ }
+
+ xids[nxids++] = xid;
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ changes_filename(path, subid, xid);
+
+ elog(DEBUG1, "opening file '%s' for streamed changes", path);
+
+ /*
+ * If this is the first streamed segment, the file must not exist, so
+ * make sure we're the ones creating it. Otherwise just open the file
+ * for writing, in append mode.
+ */
+ if (first_segment)
+ flags = (O_WRONLY | O_CREAT | O_EXCL | PG_BINARY);
+ else
+ flags = (O_WRONLY | O_APPEND | PG_BINARY);
+
+ stream_fd = OpenTransientFile(path, flags);
+
+ if (stream_fd < 0)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not open file \"%s\": %m",
+ path)));
+}
+
+/*
+ * stream_close_file
+ * Close the currently open file with streamed changes.
+ *
+ * This can only be called at the beginning of a "streaming" block, i.e.
+ * between stream_start/stream_stop messages from the upstream.
+ */
+static void
+stream_close_file(void)
+{
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ CloseTransientFile(stream_fd);
+
+ stream_xid = InvalidTransactionId;
+ stream_fd = -1;
+}
+
+/*
+ * stream_write_change
+ * Serialize a change to a file for the current toplevel transaction.
+ *
+ * The change is serialized in a simple format, with length (not including
+ * the length), action code (identifying the message type) and message
+ * contents (without the subxact TransactionId value).
+ *
+ * XXX The subxact file includes CRC32C of the contents. Maybe we should
+ * include something like that here too, but doing so will not be as
+ * straighforward, because we write the file in chunks.
+ */
+static void
+stream_write_change(char action, StringInfo s)
+{
+ int len;
+
+ Assert(in_streamed_transaction);
+ Assert(TransactionIdIsValid(stream_xid));
+ Assert(stream_fd != -1);
+
+ /* total on-disk size, including the action type character */
+ len = (s->len - s->cursor) + sizeof(char);
+
+ pgstat_report_wait_start(WAIT_EVENT_LOGICAL_CHANGES_WRITE);
+
+ /* first write the size */
+ if (write(stream_fd, &len, sizeof(len)) != sizeof(len))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* then the action */
+ if (write(stream_fd, &action, sizeof(action)) != sizeof(action))
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ /* and finally the remaining part of the buffer (after the XID) */
+ len = (s->len - s->cursor);
+
+ if (write(stream_fd, &s->data[s->cursor], len) != len)
+ ereport(ERROR,
+ (errcode_for_file_access(),
+ errmsg("could not serialize streamed change to file: %m")));
+
+ pgstat_report_wait_end();
+}
+
/* Logical Replication Apply worker entry point */
void
ApplyWorkerMain(Datum main_arg)
@@ -2106,6 +3140,7 @@ ApplyWorkerMain(Datum main_arg)
options.slotname = myslotname;
options.proto.logical.proto_version = LOGICALREP_PROTO_VERSION_NUM;
options.proto.logical.publication_names = MySubscription->publications;
+ options.proto.logical.streaming = MySubscription->stream;
/* Start normal logical streaming replication. */
walrcv_startstreaming(wrconn, &options);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 77b85fc..811706a 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -46,29 +46,54 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferChange *change);
static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,
RepOriginId origin_id);
+static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn);
+
+static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+
+static void pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
+
+static void pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn);
static bool publications_valid;
+static bool in_streaming;
static List *LoadPublications(List *pubnames);
static void publication_invalidation_cb(Datum arg, int cacheid,
uint32 hashvalue);
-static void send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx);
+static void send_relation_and_attrs(Relation relation, TransactionId xid,
+ LogicalDecodingContext *ctx);
/*
* Entry in the map used to remember which relation schemas we sent.
*
+ * The schema_sent flag determines if the current schema record was already
+ * sent to the subscriber (in which case we don't need to send it again).
+ *
+ * The schema cache on downstream is however updated only at commit time,
+ * and with streamed transactions the commit order may be different from
+ * the order the transactions are sent in. So streamed trasactions are
+ * handled separately by using schema_sent flag in ReorderBufferTXN.
+ *
* For partitions, 'pubactions' considers not only the table's own
* publications, but also those of all of its ancestors.
*/
typedef struct RelationSyncEntry
{
Oid relid; /* relation oid */
-
+ TransactionId xid; /* transaction that created the record */
/*
* Did we send the schema? If ancestor relid is set, its schema must also
* have been sent for this to be true.
*/
bool schema_sent;
+ List *streamed_txns; /* streamed toplevel transactions with this
+ * schema */
bool replicate_valid;
PublicationActions pubactions;
@@ -94,11 +119,17 @@ typedef struct RelationSyncEntry
static HTAB *RelationSyncCache = NULL;
static void init_rel_sync_cache(MemoryContext decoding_context);
+static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
uint32 hashvalue);
+static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
+ TransactionId xid);
+
/*
* Specify output plugin callbacks
*/
@@ -114,15 +145,24 @@ _PG_output_plugin_init(OutputPluginCallbacks *cb)
cb->commit_cb = pgoutput_commit_txn;
cb->filter_by_origin_cb = pgoutput_origin_filter;
cb->shutdown_cb = pgoutput_shutdown;
+
+ /* transaction streaming */
+ cb->stream_change_cb = pgoutput_change;
+ cb->stream_truncate_cb = pgoutput_truncate;
+ cb->stream_abort_cb = pgoutput_stream_abort;
+ cb->stream_commit_cb = pgoutput_stream_commit;
+ cb->stream_start_cb = pgoutput_stream_start;
+ cb->stream_stop_cb = pgoutput_stream_stop;
}
static void
parse_output_parameters(List *options, uint32 *protocol_version,
- List **publication_names)
+ List **publication_names, bool *enable_streaming)
{
ListCell *lc;
bool protocol_version_given = false;
bool publication_names_given = false;
+ bool streaming_given = false;
foreach(lc, options)
{
@@ -168,6 +208,23 @@ parse_output_parameters(List *options, uint32 *protocol_version,
(errcode(ERRCODE_INVALID_NAME),
errmsg("invalid publication_names syntax")));
}
+ else if (strcmp(defel->defname, "streaming") == 0)
+ {
+ if (streaming_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ streaming_given = true;
+
+ /* the value must be on/off */
+ if (strcmp(strVal(defel->arg), "on") && strcmp(strVal(defel->arg), "off"))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("invalid streaming value")));
+
+ /* enable streaming if it's 'on' */
+ *enable_streaming = (strcmp(strVal(defel->arg), "on") == 0);
+ }
else
elog(ERROR, "unrecognized pgoutput option: %s", defel->defname);
}
@@ -180,6 +237,7 @@ static void
pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
bool is_init)
{
+ bool enable_streaming = false;
PGOutputData *data = palloc0(sizeof(PGOutputData));
/* Create our memory context for private allocations. */
@@ -202,7 +260,8 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
/* Parse the params and ERROR if we see any we don't recognize */
parse_output_parameters(ctx->output_plugin_options,
&data->protocol_version,
- &data->publication_names);
+ &data->publication_names,
+ &enable_streaming);
/* Check if we support requested protocol */
if (data->protocol_version > LOGICALREP_PROTO_VERSION_NUM)
@@ -222,6 +281,27 @@ pgoutput_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("publication_names parameter missing")));
+ /*
+ * Decide whether to enable streaming. It is disabled by default, in
+ * which case we just update the flag in decoding context. Otherwise
+ * we only allow it with sufficient version of the protocol, and when
+ * the output plugin supports it.
+ */
+ if (!enable_streaming)
+ ctx->streaming = false;
+ else if (data->protocol_version < LOGICALREP_PROTO_STREAM_VERSION_NUM)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("requested proto_version=%d does not support streaming, need %d or higher",
+ data->protocol_version, LOGICALREP_PROTO_STREAM_VERSION_NUM)));
+ else if (!ctx->streaming)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("streaming requested, but not supported by output plugin")));
+
+ /* Also remember we're currently not streaming any transaction. */
+ in_streaming = false;
+
/* Init publication state. */
data->publications = NIL;
publications_valid = false;
@@ -290,9 +370,48 @@ pgoutput_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
*/
static void
maybe_send_schema(LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn, ReorderBufferChange *change,
Relation relation, RelationSyncEntry *relentry)
{
- if (relentry->schema_sent)
+ bool schema_sent;
+ TransactionId xid = InvalidTransactionId;
+ TransactionId topxid = InvalidTransactionId;
+
+ /*
+ * Remember XID of the (sub)transaction for the change. We don't care if
+ * it's top-level transaction or not (we have already sent that XID in
+ * start the current streaming block).
+ *
+ * If we're not in a streaming block, just use InvalidTransactionId and
+ * the write methods will not include it.
+ */
+ if (in_streaming)
+ xid = change->txn->xid;
+
+ if (change->txn->toptxn)
+ topxid = change->txn->toptxn->xid;
+ else
+ topxid = xid;
+
+ /*
+ * Do we need to send the schema? We do track streamed transactions
+ * separately, because those may not be applied later (and the regular
+ * transactions won't see their effects until then) and in an order
+ * that we don't know at this point.
+ */
+ if (in_streaming)
+ {
+ /*
+ * TOCHECK: We have to send schema after each catalog change and it may
+ * occur when streaming already started, so we have to track new catalog
+ * changes somehow.
+ */
+ schema_sent = get_schema_sent_in_streamed_txn(relentry, topxid);
+ }
+ else
+ schema_sent = relentry->schema_sent;
+
+ if (schema_sent)
return;
/* If needed, send the ancestor's schema first. */
@@ -308,19 +427,25 @@ maybe_send_schema(LogicalDecodingContext *ctx,
relentry->map = convert_tuples_by_name(CreateTupleDescCopy(indesc),
CreateTupleDescCopy(outdesc));
MemoryContextSwitchTo(oldctx);
- send_relation_and_attrs(ancestor, ctx);
+ send_relation_and_attrs(ancestor, xid, ctx);
RelationClose(ancestor);
}
- send_relation_and_attrs(relation, ctx);
- relentry->schema_sent = true;
+ send_relation_and_attrs(relation, xid, ctx);
+ relentry->xid = change->txn->xid;
+
+ if (in_streaming)
+ set_schema_sent_in_streamed_txn(relentry, topxid);
+ else
+ relentry->schema_sent = true;
}
/*
* Sends a relation
*/
static void
-send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx)
+send_relation_and_attrs(Relation relation, TransactionId xid,
+ LogicalDecodingContext *ctx)
{
TupleDesc desc = RelationGetDescr(relation);
int i;
@@ -344,17 +469,19 @@ send_relation_and_attrs(Relation relation, LogicalDecodingContext *ctx)
continue;
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_typ(ctx->out, att->atttypid);
+ logicalrep_write_typ(ctx->out, xid, att->atttypid);
OutputPluginWrite(ctx, false);
}
OutputPluginPrepareWrite(ctx, false);
- logicalrep_write_rel(ctx->out, relation);
+ logicalrep_write_rel(ctx->out, xid, relation);
OutputPluginWrite(ctx, false);
}
/*
* Sends the decoded DML over wire.
+ *
+ * XXX May be called both in streaming and non-streaming modes.
*/
static void
pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
@@ -363,6 +490,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
PGOutputData *data = (PGOutputData *) ctx->output_plugin_private;
MemoryContext old;
RelationSyncEntry *relentry;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
if (!is_publishable_relation(relation))
return;
@@ -391,7 +522,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
/* Avoid leaking memory by using and resetting our own context */
old = MemoryContextSwitchTo(data->context);
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
/* Send the data */
switch (change->action)
@@ -411,7 +542,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_insert(ctx->out, relation, tuple);
+ logicalrep_write_insert(ctx->out, xid, relation, tuple);
OutputPluginWrite(ctx, true);
break;
}
@@ -435,7 +566,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_update(ctx->out, relation, oldtuple, newtuple);
+ logicalrep_write_update(ctx->out, xid, relation, oldtuple, newtuple);
OutputPluginWrite(ctx, true);
break;
}
@@ -455,7 +586,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
}
OutputPluginPrepareWrite(ctx, true);
- logicalrep_write_delete(ctx->out, relation, oldtuple);
+ logicalrep_write_delete(ctx->out, xid, relation, oldtuple);
OutputPluginWrite(ctx, true);
}
else
@@ -480,6 +611,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
int i;
int nrelids;
Oid *relids;
+ TransactionId xid = InvalidTransactionId;
+
+ if (in_streaming)
+ xid = change->txn->xid;
old = MemoryContextSwitchTo(data->context);
@@ -508,13 +643,14 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
continue;
relids[nrelids++] = relid;
- maybe_send_schema(ctx, relation, relentry);
+ maybe_send_schema(ctx, txn, change, relation, relentry);
}
if (nrelids > 0)
{
OutputPluginPrepareWrite(ctx, true);
logicalrep_write_truncate(ctx->out,
+ xid,
nrelids,
relids,
change->data.truncate.cascade,
@@ -588,6 +724,91 @@ publication_invalidation_cb(Datum arg, int cacheid, uint32 hashvalue)
}
/*
+ * Notify downstream to apply the streamed transaction (along with all
+ * it's subtransactions).
+ */
+static void
+pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr abort_lsn)
+{
+ ReorderBufferTXN *toptxn;
+
+ /*
+ * The abort should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+
+ /* determine the toplevel transaction */
+ toptxn = (txn->toptxn) ? txn->toptxn : txn;
+
+ Assert(rbtxn_is_streamed(toptxn));
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_abort(ctx->out, toptxn->xid, txn->xid);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(toptxn->xid, false);
+}
+
+/*
+ * Notify downstream to discard the streamed transaction (along with all
+ * it's subtransactions, if it's a toplevel transaction).
+ */
+static void
+pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn)
+{
+ /*
+ * The commit should happen outside streaming block, even for streamed
+ * transactions. The transaction has to be marked as streamed, though.
+ */
+ Assert(!in_streaming);
+ Assert(rbtxn_is_streamed(txn));
+
+ OutputPluginUpdateProgress(ctx);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_commit(ctx->out, txn, commit_lsn);
+ OutputPluginWrite(ctx, true);
+
+ cleanup_rel_sync_cache(txn->xid, true);
+}
+
+
+static void
+pgoutput_stream_start(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we can't nest streaming of transactions */
+ Assert(!in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_start(ctx->out, txn->xid, !rbtxn_is_streamed(txn));
+ OutputPluginWrite(ctx, true);
+
+ /* we're streaming a chunk of transaction now */
+ in_streaming = true;
+}
+
+static void
+pgoutput_stream_stop(struct LogicalDecodingContext *ctx,
+ ReorderBufferTXN *txn)
+{
+ /* we should be streaming a trasanction */
+ Assert(in_streaming);
+
+ OutputPluginPrepareWrite(ctx, true);
+ logicalrep_write_stream_stop(ctx->out);
+ OutputPluginWrite(ctx, true);
+
+ /* we've stopped streaming a transaction */
+ in_streaming = false;
+}
+
+/*
* Initialize the relation schema sync cache for a decoding session.
*
* The hash table is destroyed at the end of a decoding session. While
@@ -624,6 +845,34 @@ init_rel_sync_cache(MemoryContext cachectx)
}
/*
+ * We expect relatively small number of streamed transactions.
+ */
+static bool
+get_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ ListCell *lc;
+ foreach (lc, entry->streamed_txns)
+ {
+ if (xid == lfirst_int(lc))
+ return true;
+ }
+
+ return false;
+}
+
+static void
+set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
+{
+ MemoryContext oldctx;
+
+ oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
+ entry->streamed_txns = lappend_int(entry->streamed_txns, xid);
+
+ MemoryContextSwitchTo(oldctx);
+}
+
+/*
* Find or create entry in the relation schema cache.
*
* This looks up publications that the given relation is directly or
@@ -750,12 +999,45 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
}
if (!found)
+ {
entry->schema_sent = false;
+ entry->streamed_txns = NULL;
+ }
return entry;
}
/*
+ * Cleanup list of streamed transactions and update the schema_sent flag.
+ *
+ * When a streamed transaction commits or aborts, we need to remove the
+ * toplevel XID from the schema cache. If the transaction aborted, the
+ * subscriber will simply throw away the schema records we streamed, so
+ * we don't need to do anything else.
+ *
+ * If the transaction committed, the subscriber will update the relation
+ * cache - so tweak the schema_sent flag accordingly.
+ */
+static void
+cleanup_rel_sync_cache(TransactionId xid, bool is_commit)
+{
+ HASH_SEQ_STATUS hash_seq;
+ RelationSyncEntry *entry;
+
+ Assert(RelationSyncCache != NULL);
+
+ hash_seq_init(&hash_seq, RelationSyncCache);
+ while ((entry = hash_seq_search(&hash_seq)) != NULL)
+ {
+ if (is_commit)
+ entry->schema_sent = true;
+
+ /* Remove the xid from the schema sent list. */
+ entry->streamed_txns = list_delete_int(entry->streamed_txns, xid);
+ }
+}
+
+/*
* Relcache invalidation callback
*/
static void
@@ -790,7 +1072,11 @@ rel_sync_cache_relation_cb(Datum arg, Oid relid)
* Reset schema sent status as the relation definition may have changed.
*/
if (entry != NULL)
+ {
entry->schema_sent = false;
+ list_free(entry->streamed_txns);
+ entry->streamed_txns = NULL;
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index f776de3..9121420 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -156,6 +156,12 @@ create_logical_replication_slot(char *name, char *plugin,
NULL);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* If caller needs us to determine the decoding start point, do so now.
* This might take a while.
*/
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 0e93322..eacea12 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1004,6 +1004,12 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd)
WalSndUpdateProgress);
/*
+ * Make sure streaming is disabled here - we may have the methods,
+ * but we don't have anywhere to send the data yet.
+ */
+ ctx->streaming = false;
+
+ /*
* Signal that we don't need the timeout mechanism. We're just
* creating the replication slot and don't yet accept feedback
* messages or send keepalives. As we possibly need to wait for
diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h
index 0a756d4..617b909 100644
--- a/src/include/catalog/pg_subscription.h
+++ b/src/include/catalog/pg_subscription.h
@@ -48,6 +48,8 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW
bool subenabled; /* True if the subscription is enabled (the
* worker should be running) */
+ bool substream; /* Stream in-progress transactions. */
+
#ifdef CATALOG_VARLEN /* variable-length fields start here */
/* Connection string to the publisher */
text subconninfo BKI_FORCE_NOT_NULL;
@@ -73,6 +75,7 @@ typedef struct Subscription
char *name; /* Name of the subscription */
Oid owner; /* Oid of the subscription owner */
bool enabled; /* Indicates if the subscription is enabled */
+ bool stream; /* Allow streaming in-progress transactions. */
char *conninfo; /* Connection string to the publisher */
char *slotname; /* Name of the replication slot */
char *synccommit; /* Synchronous commit setting for worker */
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index b8041d9..3b3e1fd 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -981,7 +981,11 @@ typedef enum
WAIT_EVENT_WAL_READ,
WAIT_EVENT_WAL_SYNC,
WAIT_EVENT_WAL_SYNC_METHOD_ASSIGN,
- WAIT_EVENT_WAL_WRITE
+ WAIT_EVENT_WAL_WRITE,
+ WAIT_EVENT_LOGICAL_CHANGES_READ,
+ WAIT_EVENT_LOGICAL_CHANGES_WRITE,
+ WAIT_EVENT_LOGICAL_SUBXACT_READ,
+ WAIT_EVENT_LOGICAL_SUBXACT_WRITE
} WaitEventIO;
/* ----------
diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 4860561..89158ed 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -23,9 +23,13 @@
* we can support. LOGICALREP_PROTO_MIN_VERSION_NUM is the oldest version we
* have backwards compatibility for. The client requests protocol version at
* connect time.
+ *
+ * LOGICALREP_PROTO_STREAM_VERSION_NUM is the minimum protocol version with
+ * support for streaming large transactions.
*/
#define LOGICALREP_PROTO_MIN_VERSION_NUM 1
-#define LOGICALREP_PROTO_VERSION_NUM 1
+#define LOGICALREP_PROTO_STREAM_VERSION_NUM 2
+#define LOGICALREP_PROTO_VERSION_NUM 2
/* Tuple coming via logical replication. */
typedef struct LogicalRepTupleData
@@ -86,25 +90,45 @@ extern void logicalrep_read_commit(StringInfo in,
extern void logicalrep_write_origin(StringInfo out, const char *origin,
XLogRecPtr origin_lsn);
extern char *logicalrep_read_origin(StringInfo in, XLogRecPtr *origin_lsn);
-extern void logicalrep_write_insert(StringInfo out, Relation rel,
- HeapTuple newtuple);
+extern void logicalrep_write_insert(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_insert(StringInfo in, LogicalRepTupleData *newtup);
-extern void logicalrep_write_update(StringInfo out, Relation rel, HeapTuple oldtuple,
+extern void logicalrep_write_update(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple,
HeapTuple newtuple);
extern LogicalRepRelId logicalrep_read_update(StringInfo in,
bool *has_oldtuple, LogicalRepTupleData *oldtup,
LogicalRepTupleData *newtup);
-extern void logicalrep_write_delete(StringInfo out, Relation rel,
- HeapTuple oldtuple);
+extern void logicalrep_write_delete(StringInfo out, TransactionId xid,
+ Relation rel, HeapTuple oldtuple);
extern LogicalRepRelId logicalrep_read_delete(StringInfo in,
LogicalRepTupleData *oldtup);
-extern void logicalrep_write_truncate(StringInfo out, int nrelids, Oid relids[],
+extern void logicalrep_write_truncate(StringInfo out, TransactionId xid,
+ int nrelids, Oid relids[],
bool cascade, bool restart_seqs);
extern List *logicalrep_read_truncate(StringInfo in,
bool *cascade, bool *restart_seqs);
-extern void logicalrep_write_rel(StringInfo out, Relation rel);
+extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel);
extern LogicalRepRelation *logicalrep_read_rel(StringInfo in);
-extern void logicalrep_write_typ(StringInfo out, Oid typoid);
+extern void logicalrep_write_typ(StringInfo out, TransactionId xid, Oid typoid);
extern void logicalrep_read_typ(StringInfo out, LogicalRepTyp *ltyp);
+extern void logicalrep_write_stream_start(StringInfo out, TransactionId xid,
+ bool first_segment);
+extern TransactionId logicalrep_read_stream_start(StringInfo in,
+ bool *first_segment);
+
+extern void logicalrep_write_stream_stop(StringInfo out);
+extern TransactionId logicalrep_read_stream_stop(StringInfo in);
+
+extern void logicalrep_write_stream_commit(StringInfo out, ReorderBufferTXN *txn,
+ XLogRecPtr commit_lsn);
+extern TransactionId logicalrep_read_stream_commit(StringInfo out,
+ LogicalRepCommitData *commit_data);
+
+extern void logicalrep_write_stream_abort(StringInfo out, TransactionId xid,
+ TransactionId subxid);
+extern void logicalrep_read_stream_abort(StringInfo in, TransactionId *xid,
+ TransactionId *subxid);
+
#endif /* LOGICAL_PROTO_H */
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index f1aa6e9..70d39f8 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -177,6 +177,7 @@ typedef struct
{
uint32 proto_version; /* Logical protocol version */
List *publication_names; /* String list of publications */
+ bool streaming; /* Streaming of large transactions */
} logical;
} proto;
} WalRcvStreamOptions;
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
new file mode 100644
index 0000000..2f01133
--- /dev/null
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -0,0 +1,86 @@
+# Test streaming of simple large transaction
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rows to exceed the 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3, 5000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(3334|3334|3334), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
new file mode 100644
index 0000000..d2ae385
--- /dev/null
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -0,0 +1,102 @@
+# Test streaming of large transaction containing large subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 3;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c timestamptz DEFAULT now(), d bigint DEFAULT 999)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|2|2), 'check initial data was copied to subscriber');
+
+# Insert, update and delete enough rowsto exceed 64kB limit.
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series( 3, 500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501, 1000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001, 1500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501, 2000) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+SAVEPOINT s4;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001, 2500) s(i);
+UPDATE test_tab SET b = md5(b) WHERE mod(a,2) = 0;
+DELETE FROM test_tab WHERE mod(a,3) = 0;
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain local defaults');
+
+# Change the local values of the extra columns on the subscriber,
+# update publisher, and check that subscriber retains the expected
+# values
+$node_subscriber->safe_psql('postgres', "UPDATE test_tab SET c = 'epoch'::timestamptz + 987654321 * interval '1s'");
+$node_publisher->safe_psql('postgres', "UPDATE test_tab SET b = md5(a::text)");
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(extract(epoch from c) = 987654321), count(d = 999) FROM test_tab");
+is($result, qq(1667|1667|1667), 'check extra columns contain locally changed data');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
new file mode 100644
index 0000000..0da39a1
--- /dev/null
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -0,0 +1,95 @@
+# Test streaming of large transaction with DDL and subtransactions
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d = 999) FROM test_tab");
+is($result, qq(2|0|0), 'check initial data was copied to subscriber');
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (3, md5(3::text));
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (4, md5(4::text), -4);
+COMMIT;
+});
+
+# large (streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(5, 1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001, 2000) s(i);
+COMMIT;
+});
+
+# a small (non-streamed) transaction with DDL and DML
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab VALUES (2001, md5(2001::text), -2001, 2*2001);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s1;
+INSERT INTO test_tab VALUES (2002, md5(2002::text), -2002, 2*2002, -3*2002);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c), count(d), count(e) FROM test_tab");
+is($result, qq(2002|1999|1002|1), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
new file mode 100644
index 0000000..402df30
--- /dev/null
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -0,0 +1,82 @@
+# Test streaming of large transaction containing multiple subtransactions and rollbacks
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(501,1000) s(i);
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1001,1500) s(i);
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(1501,2000) s(i);
+ROLLBACK TO s2;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2001,2500) s(i);
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(2501,3000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|0), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
new file mode 100644
index 0000000..becbdd0
--- /dev/null
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -0,0 +1,84 @@
+# Test behavior with streaming transaction exceeding logical_decoding_work_mem
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 2;
+
+sub wait_for_caught_up
+{
+ my ($node, $appname) = @_;
+
+ $node->poll_query_until('postgres',
+"SELECT pg_current_wal_lsn() <= replay_lsn FROM pg_stat_replication WHERE application_name = '$appname';"
+ ) or die "Timed out while waiting for subscriber to catch up";
+}
+
+# Create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'logical_decoding_work_mem = 64kB');
+$node_publisher->start;
+
+# Create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# Create some preexisting content on publisher
+$node_publisher->safe_psql('postgres',
+ "CREATE TABLE test_tab (a int primary key, b varchar)");
+$node_publisher->safe_psql('postgres',
+ "INSERT INTO test_tab VALUES (1, 'foo'), (2, 'bar')");
+
+# Setup structure on subscriber
+$node_subscriber->safe_psql('postgres', "CREATE TABLE test_tab (a int primary key, b text, c INT, d INT, e INT)");
+
+# Setup logical replication
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+$node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE test_tab");
+
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+);
+
+wait_for_caught_up($node_publisher, $appname);
+
+# Also wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+ or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(2|0), 'check initial data was copied to subscriber');
+
+# large (streamed) transaction with DDL, DML and ROLLBACKs
+$node_publisher->safe_psql('postgres', q{
+BEGIN;
+INSERT INTO test_tab SELECT i, md5(i::text) FROM generate_series(3,500) s(i);
+ALTER TABLE test_tab ADD COLUMN c INT;
+SAVEPOINT s1;
+INSERT INTO test_tab SELECT i, md5(i::text), -i FROM generate_series(501,1000) s(i);
+ALTER TABLE test_tab ADD COLUMN d INT;
+SAVEPOINT s2;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i FROM generate_series(1001,1500) s(i);
+ALTER TABLE test_tab ADD COLUMN e INT;
+SAVEPOINT s3;
+INSERT INTO test_tab SELECT i, md5(i::text), -i, 2*i, -3*i FROM generate_series(1501,2000) s(i);
+ALTER TABLE test_tab DROP COLUMN c;
+ROLLBACK TO s1;
+INSERT INTO test_tab SELECT i, md5(i::text), i FROM generate_series(501,1000) s(i);
+COMMIT;
+});
+
+wait_for_caught_up($node_publisher, $appname);
+
+$result =
+ $node_subscriber->safe_psql('postgres', "SELECT count(*), count(c) FROM test_tab");
+is($result, qq(1000|500), 'check extra columns contain local defaults');
+
+$node_subscriber->stop;
+$node_publisher->stop;
--
1.8.3.1
v16-0007-Track-statistics-for-streaming.patch (application/octet-stream, 11.7K) download
From 06ed149a64907c0814b305936cca83b66ae89ff1 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Thu, 2 Apr 2020 13:19:29 +0530
Subject: [PATCH v16 07/12] Track statistics for streaming
---
doc/src/sgml/monitoring.sgml | 25 +++++++++++++++++++
src/backend/catalog/system_views.sql | 5 +++-
src/backend/replication/logical/reorderbuffer.c | 13 ++++++++++
src/backend/replication/walsender.c | 32 +++++++++++++++++++++----
src/include/catalog/pg_proc.dat | 6 ++---
src/include/replication/reorderbuffer.h | 13 ++++++----
src/include/replication/walsender_private.h | 5 ++++
src/test/regress/expected/rules.out | 7 ++++--
8 files changed, 91 insertions(+), 15 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 6562cc4..d8bf587 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2063,6 +2063,31 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
may get spilled repeatedly, and this counter gets incremented on every
such invocation.</entry>
</row>
+ <row>
+ <entry><structfield>stream_txns</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of in-progress transactions streamed to subscriber after
+ memory used by logical decoding exceeds <literal>logical_work_mem</literal>.
+ Streaming only works with toplevel transactions (subtransactions can't
+ be streamed independently), so the counter does not get incremented for
+ subtransactions.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_count</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Number of times in-progress transactions were streamed to subscriber.
+ Transactions may get streamed repeatedly, and this counter gets incremented
+ on every such invocation.
+ </entry>
+ </row>
+ <row>
+ <entry><structfield>stream_bytes</structfield></entry>
+ <entry><type>integer</type></entry>
+ <entry>Amount of decoded in-progress transaction data streamed to subscriber.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index d406ea8..65d650d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -788,7 +788,10 @@ CREATE VIEW pg_stat_replication AS
W.reply_time,
W.spill_txns,
W.spill_count,
- W.spill_bytes
+ W.spill_bytes,
+ W.stream_txns,
+ W.stream_count,
+ W.stream_bytes
FROM pg_stat_get_activity(NULL) AS S
JOIN pg_stat_get_wal_senders() AS W ON (S.pid = W.pid)
LEFT JOIN pg_authid AS U ON (S.usesysid = U.oid);
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 1868ab2..e207f90 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -331,6 +331,10 @@ ReorderBufferAllocate(void)
buffer->spillTxns = 0;
buffer->spillBytes = 0;
+ buffer->streamCount = 0;
+ buffer->streamTxns = 0;
+ buffer->streamBytes = 0;
+
buffer->current_restart_decoding_lsn = InvalidXLogRecPtr;
dlist_init(&buffer->toplevel_by_lsn);
@@ -3287,6 +3291,15 @@ ReorderBufferStreamTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
ReorderBufferProcessTXN(rb, txn, InvalidXLogRecPtr, snapshot_now,
command_id, true);
+ /*
+ * Update the stream statistics.
+ */
+ rb->streamCount += 1;
+ rb->streamBytes += txn->size;
+
+ /* Don't consider already streamed transaction. */
+ rb->streamTxns += (rbtxn_is_streamed(txn)) ? 0 : 1;
+
Assert(dlist_is_empty(&txn->changes));
Assert(txn->nentries == 0);
Assert(txn->nentries_mem == 0);
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index eacea12..d0028d9 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -1333,7 +1333,7 @@ WalSndWriteData(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid,
* LogicalDecodingContext 'update_progress' callback.
*
* Write the current position to the lag tracker (see XLogSendPhysical),
- * and update the spill statistics.
+ * and update the spill/stream statistics.
*/
static void
WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid)
@@ -1354,7 +1354,8 @@ WalSndUpdateProgress(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId
sendTime = now;
/*
- * Update statistics about transactions that spilled to disk.
+ * Update statistics about transactions that spilled to disk or streamed to
+ * subscriber (before being committed).
*/
UpdateSpillStats(ctx);
}
@@ -2396,6 +2397,9 @@ InitWalSenderSlot(void)
walsnd->spillTxns = 0;
walsnd->spillCount = 0;
walsnd->spillBytes = 0;
+ walsnd->streamTxns = 0;
+ walsnd->streamCount = 0;
+ walsnd->streamBytes = 0;
SpinLockRelease(&walsnd->mutex);
/* don't need the lock anymore */
MyWalSnd = (WalSnd *) walsnd;
@@ -3237,7 +3241,7 @@ offset_to_interval(TimeOffset offset)
Datum
pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
{
-#define PG_STAT_GET_WAL_SENDERS_COLS 15
+#define PG_STAT_GET_WAL_SENDERS_COLS 18
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -3295,6 +3299,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
int64 spillCount;
int64 spillBytes;
bool is_sync_standby;
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
Datum values[PG_STAT_GET_WAL_SENDERS_COLS];
bool nulls[PG_STAT_GET_WAL_SENDERS_COLS];
int j;
@@ -3320,6 +3327,9 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
spillTxns = walsnd->spillTxns;
spillCount = walsnd->spillCount;
spillBytes = walsnd->spillBytes;
+ streamTxns = walsnd->streamTxns;
+ streamCount = walsnd->streamCount;
+ streamBytes = walsnd->streamBytes;
SpinLockRelease(&walsnd->mutex);
/*
@@ -3422,6 +3432,11 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
values[12] = Int64GetDatum(spillTxns);
values[13] = Int64GetDatum(spillCount);
values[14] = Int64GetDatum(spillBytes);
+
+ /* stream over-sized transactions */
+ values[15] = Int64GetDatum(streamTxns);
+ values[16] = Int64GetDatum(streamCount);
+ values[17] = Int64GetDatum(streamBytes);
}
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
@@ -3670,11 +3685,18 @@ UpdateSpillStats(LogicalDecodingContext *ctx)
MyWalSnd->spillCount = rb->spillCount;
MyWalSnd->spillBytes = rb->spillBytes;
- elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld",
+ MyWalSnd->streamTxns = rb->streamTxns;
+ MyWalSnd->streamCount = rb->streamCount;
+ MyWalSnd->streamBytes = rb->streamBytes;
+
+ elog(DEBUG2, "UpdateSpillStats: updating stats %p %lld %lld %lld %lld %lld %lld",
rb,
(long long) rb->spillTxns,
(long long) rb->spillCount,
- (long long) rb->spillBytes);
+ (long long) rb->spillBytes,
+ (long long) rb->streamTxns,
+ (long long) rb->streamCount,
+ (long long) rb->streamBytes);
SpinLockRelease(&MyWalSnd->mutex);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 4bce3ad..9fb1ffe 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5237,9 +5237,9 @@
proname => 'pg_stat_get_wal_senders', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', proparallel => 'r',
prorettype => 'record', proargtypes => '',
- proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes}',
+ proallargtypes => '{int4,text,pg_lsn,pg_lsn,pg_lsn,pg_lsn,interval,interval,interval,int4,text,timestamptz,int8,int8,int8,int8,int8,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid,state,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priority,sync_state,reply_time,spill_txns,spill_count,spill_bytes,stream_txns,stream_count,stream_bytes}',
prosrc => 'pg_stat_get_wal_senders' },
{ oid => '3317', descr => 'statistics: information about WAL receiver',
proname => 'pg_stat_get_wal_receiver', proisstrict => 'f', provolatile => 's',
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6d65986..603f325 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -517,15 +517,20 @@ struct ReorderBuffer
Size size;
/*
- * Statistics about transactions spilled to disk.
+ * Statistics about transactions streamed or spilled to disk.
*
- * A single transaction may be spilled repeatedly, which is why we keep
- * two different counters. For spilling, the transaction counter includes
- * both toplevel transactions and subtransactions.
+ * A single transaction may be streamed/spilled repeatedly, which is
+ * why we keep two different counters. For spilling, the transaction
+ * counter includes both toplevel transactions and subtransactions.
+ * For streaming, it only includes toplevel transactions (we never
+ * stream individual subtransactions).
*/
int64 spillCount; /* spill-to-disk invocation counter */
int64 spillTxns; /* number of transactions spilled to disk */
int64 spillBytes; /* amount of data spilled to disk */
+ int64 streamCount; /* streaming invocation counter */
+ int64 streamTxns; /* number of transactions spilled to disk */
+ int64 streamBytes; /* amount of data streamed to subscriber */
};
diff --git a/src/include/replication/walsender_private.h b/src/include/replication/walsender_private.h
index 734acec..b997d17 100644
--- a/src/include/replication/walsender_private.h
+++ b/src/include/replication/walsender_private.h
@@ -83,6 +83,11 @@ typedef struct WalSnd
int64 spillTxns;
int64 spillCount;
int64 spillBytes;
+
+ /* Statistics for in-progress transactions streamed to subscriber. */
+ int64 streamTxns;
+ int64 streamCount;
+ int64 streamBytes;
} WalSnd;
extern WalSnd *MyWalSnd;
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index ac31840..68e2deb 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2005,9 +2005,12 @@ pg_stat_replication| SELECT s.pid,
w.reply_time,
w.spill_txns,
w.spill_count,
- w.spill_bytes
+ w.spill_bytes,
+ w.stream_txns,
+ w.stream_count,
+ w.stream_bytes
FROM ((pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, wait_event_type, wait_event, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port, backend_xid, backend_xmin, backend_type, ssl, sslversion, sslcipher, sslbits, sslcompression, ssl_client_dn, ssl_client_serial, ssl_issuer_dn, gss_auth, gss_princ, gss_enc, leader_pid)
- JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes) ON ((s.pid = w.pid)))
+ JOIN pg_stat_get_wal_senders() w(pid, state, sent_lsn, write_lsn, flush_lsn, replay_lsn, write_lag, flush_lag, replay_lag, sync_priority, sync_state, reply_time, spill_txns, spill_count, spill_bytes, stream_txns, stream_count, stream_bytes) ON ((s.pid = w.pid)))
LEFT JOIN pg_authid u ON ((s.usesysid = u.oid)));
pg_stat_slru| SELECT s.name,
s.blks_zeroed,
--
1.8.3.1
v16-0008-Enable-streaming-for-all-subscription-TAP-tests.patch (application/octet-stream, 14.7K) download
From 38e1c3a16b6e3b0be12c3e7a0ec723f0823e8bb6 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Wed, 20 Nov 2019 16:41:13 +0530
Subject: [PATCH v16 08/12] Enable streaming for all subscription TAP tests
---
src/test/subscription/t/001_rep_changes.pl | 2 +-
src/test/subscription/t/002_types.pl | 2 +-
src/test/subscription/t/003_constraints.pl | 2 +-
src/test/subscription/t/004_sync.pl | 8 ++++----
src/test/subscription/t/005_encoding.pl | 2 +-
src/test/subscription/t/006_rewrite.pl | 2 +-
src/test/subscription/t/007_ddl.pl | 2 +-
src/test/subscription/t/008_diff_schema.pl | 2 +-
src/test/subscription/t/009_matviews.pl | 2 +-
src/test/subscription/t/009_stream_simple.pl | 2 +-
src/test/subscription/t/010_stream_subxact.pl | 2 +-
src/test/subscription/t/010_truncate.pl | 6 +++---
src/test/subscription/t/011_generated.pl | 2 +-
src/test/subscription/t/011_stream_ddl.pl | 2 +-
src/test/subscription/t/012_collation.pl | 2 +-
src/test/subscription/t/012_stream_subxact_abort.pl | 2 +-
src/test/subscription/t/013_stream_subxact_ddl_abort.pl | 2 +-
src/test/subscription/t/100_bugs.pl | 2 +-
18 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index 6da7f71..086d0c7 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -70,7 +70,7 @@ $node_publisher->safe_psql('postgres',
"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub, tap_pub_ins_only WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/002_types.pl b/src/test/subscription/t/002_types.pl
index aedcab2..94c71f8 100644
--- a/src/test/subscription/t/002_types.pl
+++ b/src/test/subscription/t/002_types.pl
@@ -108,7 +108,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (slot_name = tap_sub_slot, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/003_constraints.pl b/src/test/subscription/t/003_constraints.pl
index 34ab11e..ad3ed13 100644
--- a/src/test/subscription/t/003_constraints.pl
+++ b/src/test/subscription/t/003_constraints.pl
@@ -35,7 +35,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/004_sync.pl b/src/test/subscription/t/004_sync.pl
index e111ab9..a6fae9c 100644
--- a/src/test/subscription/t/004_sync.pl
+++ b/src/test/subscription/t/004_sync.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
@@ -56,7 +56,7 @@ $node_publisher->safe_psql('postgres',
# recreate the subscription, it will try to do initial copy
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# but it will be stuck on data copy as it will fail on constraint
@@ -78,7 +78,7 @@ is($result, qq(20), 'initial data synced for second sub');
# now check another subscription for the same node pair
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false)"
+ "CREATE SUBSCRIPTION tap_sub2 CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (copy_data = false, streaming = on)"
);
# wait for it to start
@@ -100,7 +100,7 @@ $node_subscriber->safe_psql('postgres', "DELETE FROM tab_rep;");
# recreate the subscription again
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
# and wait for data sync to finish again
diff --git a/src/test/subscription/t/005_encoding.pl b/src/test/subscription/t/005_encoding.pl
index aec7a17..202871a 100644
--- a/src/test/subscription/t/005_encoding.pl
+++ b/src/test/subscription/t/005_encoding.pl
@@ -26,7 +26,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/006_rewrite.pl b/src/test/subscription/t/006_rewrite.pl
index c6cda10..70c86b2 100644
--- a/src/test/subscription/t/006_rewrite.pl
+++ b/src/test/subscription/t/006_rewrite.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/007_ddl.pl b/src/test/subscription/t/007_ddl.pl
index 7fe6cc6..f9c8d1d 100644
--- a/src/test/subscription/t/007_ddl.pl
+++ b/src/test/subscription/t/007_ddl.pl
@@ -22,7 +22,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->wait_for_catchup('mysub');
diff --git a/src/test/subscription/t/008_diff_schema.pl b/src/test/subscription/t/008_diff_schema.pl
index 81520a7..0c9c6b3 100644
--- a/src/test/subscription/t/008_diff_schema.pl
+++ b/src/test/subscription/t/008_diff_schema.pl
@@ -32,7 +32,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION tap_pub FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub"
+ "CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr' PUBLICATION tap_pub WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('tap_sub');
diff --git a/src/test/subscription/t/009_matviews.pl b/src/test/subscription/t/009_matviews.pl
index 7afc7bd..21f50c7 100644
--- a/src/test/subscription/t/009_matviews.pl
+++ b/src/test/subscription/t/009_matviews.pl
@@ -18,7 +18,7 @@ my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION mypub FOR ALL TABLES;");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub;"
+ "CREATE SUBSCRIPTION mysub CONNECTION '$publisher_connstr' PUBLICATION mypub WITH (streaming = on);"
);
$node_publisher->safe_psql('postgres',
diff --git a/src/test/subscription/t/009_stream_simple.pl b/src/test/subscription/t/009_stream_simple.pl
index 2f01133..30561d8 100644
--- a/src/test/subscription/t/009_stream_simple.pl
+++ b/src/test/subscription/t/009_stream_simple.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_stream_subxact.pl b/src/test/subscription/t/010_stream_subxact.pl
index d2ae385..9a6bac6 100644
--- a/src/test/subscription/t/010_stream_subxact.pl
+++ b/src/test/subscription/t/010_stream_subxact.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/010_truncate.pl b/src/test/subscription/t/010_truncate.pl
index be2c0bd..ed56fbf 100644
--- a/src/test/subscription/t/010_truncate.pl
+++ b/src/test/subscription/t/010_truncate.pl
@@ -52,13 +52,13 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub3 FOR TABLE tab3, tab4");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2"
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (streaming = on)"
);
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3"
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_generated.pl b/src/test/subscription/t/011_generated.pl
index f35d1cb..4df1dde 100644
--- a/src/test/subscription/t/011_generated.pl
+++ b/src/test/subscription/t/011_generated.pl
@@ -33,7 +33,7 @@ $node_publisher->safe_psql('postgres',
$node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
# Wait for initial sync of all subscriptions
diff --git a/src/test/subscription/t/011_stream_ddl.pl b/src/test/subscription/t/011_stream_ddl.pl
index 0da39a1..c3caff6 100644
--- a/src/test/subscription/t/011_stream_ddl.pl
+++ b/src/test/subscription/t/011_stream_ddl.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/012_collation.pl b/src/test/subscription/t/012_collation.pl
index 4bfcef7..c62eb52 100644
--- a/src/test/subscription/t/012_collation.pl
+++ b/src/test/subscription/t/012_collation.pl
@@ -80,7 +80,7 @@ $node_publisher->safe_psql('postgres',
q{CREATE PUBLICATION pub1 FOR ALL TABLES});
$node_subscriber->safe_psql('postgres',
- qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false)}
+ qq{CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (copy_data = false, streaming = on)}
);
$node_publisher->wait_for_catchup('sub1');
diff --git a/src/test/subscription/t/012_stream_subxact_abort.pl b/src/test/subscription/t/012_stream_subxact_abort.pl
index 402df30..2be7542 100644
--- a/src/test/subscription/t/012_stream_subxact_abort.pl
+++ b/src/test/subscription/t/012_stream_subxact_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
index becbdd0..2da9607 100644
--- a/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
+++ b/src/test/subscription/t/013_stream_subxact_ddl_abort.pl
@@ -40,7 +40,7 @@ $node_publisher->safe_psql('postgres', "CREATE PUBLICATION tap_pub FOR TABLE tes
my $appname = 'tap_sub';
$node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub WITH (streaming = on)"
);
wait_for_caught_up($node_publisher, $appname);
diff --git a/src/test/subscription/t/100_bugs.pl b/src/test/subscription/t/100_bugs.pl
index 366a7a9..96ffc09 100644
--- a/src/test/subscription/t/100_bugs.pl
+++ b/src/test/subscription/t/100_bugs.pl
@@ -53,7 +53,7 @@ $node_publisher->safe_psql('postgres',
"CREATE PUBLICATION pub1 FOR ALL TABLES");
$node_subscriber->safe_psql('postgres',
- "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1"
+ "CREATE SUBSCRIPTION sub1 CONNECTION '$publisher_connstr' PUBLICATION pub1 WITH (streaming = on)"
);
$node_publisher->wait_for_catchup('sub1');
--
1.8.3.1
v16-0010-Bugfix-handling-of-incomplete-toast-tuple.patch (application/octet-stream, 15.3K) download
From 1dc90a93222e4cef0f648f63d5d0efc8888f9c4f Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilip.kumar@enterprisedb.com>
Date: Fri, 28 Feb 2020 11:07:46 +0530
Subject: [PATCH v16 10/12] Bugfix handling of incomplete toast tuple
---
src/backend/access/heap/heapam.c | 3 +
src/backend/replication/logical/decode.c | 17 ++-
src/backend/replication/logical/reorderbuffer.c | 182 +++++++++++++++---------
src/include/access/heapam_xlog.h | 1 +
src/include/replication/reorderbuffer.h | 24 +++-
5 files changed, 147 insertions(+), 80 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 84884a4..18b0bad 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -1978,6 +1978,9 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid,
{
xlrec.flags |= XLH_INSERT_CONTAINS_NEW_TUPLE;
bufflags |= REGBUF_KEEP_DATA;
+
+ if (IsToastRelation(relation))
+ xlrec.flags |= XLH_INSERT_ON_TOAST_RELATION;
}
XLogBeginInsert();
diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c
index 69c1f45..c841687 100644
--- a/src/backend/replication/logical/decode.c
+++ b/src/backend/replication/logical/decode.c
@@ -727,7 +727,9 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change,
+ xlrec->flags & XLH_INSERT_ON_TOAST_RELATION);
}
/*
@@ -794,7 +796,8 @@ DecodeUpdate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -851,7 +854,8 @@ DecodeDelete(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
/*
@@ -887,7 +891,7 @@ DecodeTruncate(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
memcpy(change->data.truncate.relids, xlrec->relids,
xlrec->nrelids * sizeof(Oid));
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
}
/*
@@ -987,7 +991,7 @@ DecodeMultiInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = false;
ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r),
- buf->origptr, change);
+ buf->origptr, change, false);
/* move to the next xl_multi_insert_tuple entry */
data += datalen;
@@ -1025,7 +1029,8 @@ DecodeSpecConfirm(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
change->data.tp.clear_toast_afterwards = true;
- ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr, change);
+ ReorderBufferQueueChange(ctx->reorder, XLogRecGetXid(r), buf->origptr,
+ change, false);
}
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index e207f90..eed9a50 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -654,11 +654,14 @@ ReorderBufferTXNByXid(ReorderBuffer *rb, TransactionId xid, bool create,
*/
void
ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
- ReorderBufferChange *change)
+ ReorderBufferChange *change, bool toast_insert)
{
- ReorderBufferTXN *txn;
+ ReorderBufferTXN *txn, *toptxn;
+ bool can_stream = false;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ toptxn = txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+ if (txn->toptxn != NULL)
+ toptxn = txn->toptxn;
change->lsn = lsn;
change->txn = txn;
@@ -668,9 +671,50 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
txn->nentries++;
txn->nentries_mem++;
+ /*
+ * If this is a toast insert then set the corresponding bit. Otherwise, if
+ * we have toast insert bit set and this is insert/update then clear the
+ * bit.
+ */
+ if (toast_insert)
+ toptxn->txn_flags |= RBTXN_HAS_TOAST_INSERT;
+ else if (rbtxn_has_toast_insert(txn) &&
+ ((change->action == REORDER_BUFFER_CHANGE_INSERT) ||
+ (change->action == REORDER_BUFFER_CHANGE_UPDATE)))
+ {
+ toptxn->txn_flags &= ~RBTXN_HAS_TOAST_INSERT;
+ can_stream = true;
+ }
+
+ /*
+ * If this is a speculative insert then set the corresponding bit.
+ * Otherwise, if we have speculative insert bit set and this is spec confirm
+ * record then clear the bit.
+ */
+ if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT)
+ toptxn->txn_flags |= RBTXN_HAS_SPEC_INSERT;
+ else if (change->action == REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM)
+ {
+ toptxn->txn_flags &= ~RBTXN_HAS_SPEC_INSERT;
+ can_stream = true;
+ }
+
/* update memory accounting information */
ReorderBufferChangeMemoryUpdate(rb, change, true);
+ /*
+ * If streaming is enable and we have serialized this transaction because
+ * it had incomplete tuple. So if now we have got the complete tuple we
+ * can stream it.
+ */
+ if (ReorderBufferCanStream(rb) && can_stream && rbtxn_is_serialized(toptxn)
+ && !rbtxn_has_toast_insert(txn) && !rbtxn_has_spec_insert(txn))
+ {
+ ReorderBufferStreamTXN(rb, toptxn);
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
+ }
+
/* check the memory limits and evict something if needed */
ReorderBufferCheckMemoryLimit(rb);
}
@@ -700,7 +744,7 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid,
change->data.msg.message = palloc(message_size);
memcpy(change->data.msg.message, message, message_size);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -1861,8 +1905,8 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
* disk.
*/
dlist_delete(&change->node);
- ReorderBufferToastAppendChunk(rb, txn, relation,
- change);
+ ReorderBufferToastAppendChunk(rb, txn, relation,
+ change);
}
change_done:
@@ -2461,7 +2505,7 @@ ReorderBufferAddSnapshot(ReorderBuffer *rb, TransactionId xid,
change->data.snapshot = snap;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2510,7 +2554,7 @@ ReorderBufferAddNewCommandId(ReorderBuffer *rb, TransactionId xid,
change->data.command_id = cid;
change->action = REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID;
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
}
/*
@@ -2533,6 +2577,7 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
Size sz;
ReorderBufferTXN *txn;
+ ReorderBufferTXN *toptxn = NULL;
Assert(change->txn);
@@ -2547,8 +2592,13 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
txn = change->txn;
/* if subxact, and streaming supported, use the toplevel instead */
- if (txn->toptxn && ReorderBufferCanStream(rb))
- txn = txn->toptxn;
+ if (ReorderBufferCanStream(rb))
+ {
+ if (txn->toptxn)
+ toptxn = txn->toptxn;
+ else
+ toptxn = txn;
+ }
sz = ReorderBufferChangeSize(change);
@@ -2556,12 +2606,20 @@ ReorderBufferChangeMemoryUpdate(ReorderBuffer *rb,
{
txn->size += sz;
rb->size += sz;
+
+ /* Update the total size in the top transaction. */
+ if (toptxn)
+ toptxn->total_size += sz;
}
else
{
Assert((rb->size >= sz) && (txn->size >= sz));
txn->size -= sz;
rb->size -= sz;
+
+ /* Update the total size in the top transaction. */
+ if (toptxn)
+ toptxn->total_size -= sz;
}
Assert(txn->size <= rb->size);
@@ -2622,7 +2680,7 @@ ReorderBufferAddInvalidation(ReorderBuffer *rb, TransactionId xid,
memcpy(change->data.inval.invalidations, msgs,
sizeof(SharedInvalidationMessage) * nmsgs);
- ReorderBufferQueueChange(rb, xid, lsn, change);
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
MemoryContextSwitchTo(oldcontext);
}
@@ -2809,15 +2867,16 @@ ReorderBufferLargestTopTXN(ReorderBuffer *rb)
txn = dlist_container(ReorderBufferTXN, node, iter.cur);
- /* if the current transaction is larger, remember it */
- if ((!largest) || (txn->size > largest->size))
- largest = txn;
+ /*
+ * if the current transaction is larger and doesn't have incomplete data
+ * remember it.
+ */
+ if (((!largest) || (txn->total_size > largest->total_size)) &&
+ ((txn->size > 0) && !rbtxn_has_toast_insert(txn) &&
+ !rbtxn_has_spec_insert(txn)))
+ largest = txn;
}
- Assert(largest);
- Assert(largest->size > 0);
- Assert(largest->size <= rb->size);
-
return largest;
}
@@ -2835,66 +2894,51 @@ ReorderBufferCheckMemoryLimit(ReorderBuffer *rb)
{
ReorderBufferTXN *txn;
- /* bail out if we haven't exceeded the memory limit */
- if (rb->size < logical_decoding_work_mem * 1024L)
- return;
-
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by streaming, if supported. Otherwise spill to disk.
- */
- if (ReorderBufferCanStream(rb))
+ /* Loop until we reach under the memory limit. */
+ while (rb->size >= logical_decoding_work_mem * 1024L)
{
/*
- * Pick the largest toplevel transaction and evict it from memory by
- * streaming the already decoded part.
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by streaming, if supported. Otherwise spill to disk.
*/
- txn = ReorderBufferLargestTopTXN(rb);
+ if (ReorderBufferCanStream(rb) &&
+ (txn = ReorderBufferLargestTopTXN(rb)) != NULL)
+ {
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn && !txn->toptxn);
+ Assert(txn->total_size > 0);
+ Assert(rb->size >= txn->total_size);
- /* we know there has to be one, because the size is not zero */
- Assert(txn && !txn->toptxn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ ReorderBufferStreamTXN(rb, txn);
+ }
+ else
+ {
+ /*
+ * Pick the largest transaction (or subtransaction) and evict it from
+ * memory by serializing it to disk.
+ */
+ txn = ReorderBufferLargestTXN(rb);
- ReorderBufferStreamTXN(rb, txn);
- }
- else
- {
- /*
- * Pick the largest transaction (or subtransaction) and evict it from
- * memory by serializing it to disk.
- */
- txn = ReorderBufferLargestTXN(rb);
+ /* we know there has to be one, because the size is not zero */
+ Assert(txn);
+ Assert(txn->size > 0);
+ Assert(rb->size >= txn->size);
- /* we know there has to be one, because the size is not zero */
- Assert(txn);
- Assert(txn->size > 0);
- Assert(rb->size >= txn->size);
+ ReorderBufferSerializeTXN(rb, txn);
+ }
- ReorderBufferSerializeTXN(rb, txn);
+ /*
+ * After eviction, the transaction should have no entries in memory, and
+ * should use 0 bytes for changes.
+ *
+ * XXX Checking the size is fine for both cases - spill to disk and
+ * streaming. But for streaming we should really check nentries_mem for
+ * all subtransactions too.
+ */
+ Assert(txn->size == 0);
+ Assert(txn->nentries_mem == 0);
}
- /*
- * After eviction, the transaction should have no entries in memory, and
- * should use 0 bytes for changes.
- *
- * XXX Checking the size is fine for both cases - spill to disk and
- * streaming. But for streaming we should really check nentries_mem for
- * all subtransactions too.
- */
- Assert(txn->size == 0);
- Assert(txn->nentries_mem == 0);
-
- /*
- * And furthermore, evicting the transaction should get us below the
- * memory limit again - it is not possible that we're still exceeding the
- * memory limit after evicting the transaction.
- *
- * This follows from the simple fact that the selected transaction is at
- * least as large as the most recent change (which caused us to go over
- * the memory limit). So by evicting it we're definitely back below the
- * memory limit.
- */
Assert(rb->size < logical_decoding_work_mem * 1024L);
}
diff --git a/src/include/access/heapam_xlog.h b/src/include/access/heapam_xlog.h
index 95d18cd..aa17f7d 100644
--- a/src/include/access/heapam_xlog.h
+++ b/src/include/access/heapam_xlog.h
@@ -67,6 +67,7 @@
#define XLH_INSERT_LAST_IN_MULTI (1<<1)
#define XLH_INSERT_IS_SPECULATIVE (1<<2)
#define XLH_INSERT_CONTAINS_NEW_TUPLE (1<<3)
+#define XLH_INSERT_ON_TOAST_RELATION (1<<4)
/*
* xl_heap_update flag values, 8 bits are available.
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 603f325..ba2ab71 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -172,6 +172,8 @@ typedef struct ReorderBufferChange
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
#define RBTXN_IS_STREAMED 0x0008
+#define RBTXN_HAS_TOAST_INSERT 0x0010
+#define RBTXN_HAS_SPEC_INSERT 0x0020
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -191,6 +193,17 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/* This transaction's changes has toast insert, without main table insert. */
+#define rbtxn_has_toast_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_TOAST_INSERT) != 0 \
+
+/*
+ * This transaction's changes has speculative insert, without speculative
+ * confirm.
+ */
+#define rbtxn_has_spec_insert(txn) \
+ ((txn)->txn_flags & RBTXN_HAS_SPEC_INSERT) != 0 \
+
/*
* Has this transaction been streamed to downstream?
*
@@ -199,10 +212,6 @@ typedef struct ReorderBufferChange
* which case we'd have nentries==0 for the toplevel one, which would say
* nothing about the streaming. So we maintain this flag, but only for the
* toplevel transaction.)
- *
- * Note: We never do both stream and serialize a transaction (we only spill
- * to disk when streaming is not supported by the plugin), so only one of
- * those two flags may be set at any given time.
*/
#define rbtxn_is_streamed(txn) \
( \
@@ -355,6 +364,9 @@ typedef struct ReorderBufferTXN
* Size of this transaction (changes currently in memory, in bytes).
*/
Size size;
+
+ /* Size of top-transaction including sub-transactions. */
+ Size total_size;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -545,7 +557,9 @@ void ReorderBufferReturnChange(ReorderBuffer *, ReorderBufferChange *);
Oid *ReorderBufferGetRelids(ReorderBuffer *, int nrelids);
void ReorderBufferReturnRelids(ReorderBuffer *, Oid *relids);
-void ReorderBufferQueueChange(ReorderBuffer *, TransactionId, XLogRecPtr lsn, ReorderBufferChange *);
+void ReorderBufferQueueChange(ReorderBuffer *, TransactionId,
+ XLogRecPtr lsn, ReorderBufferChange *,
+ bool incomplte_data);
void ReorderBufferQueueMessage(ReorderBuffer *, TransactionId, Snapshot snapshot, XLogRecPtr lsn,
bool transactional, const char *prefix,
Size message_size, const char *message);
--
1.8.3.1
v16-0012-Add-streaming-option-in-pg_dump.patch (application/octet-stream, 2.7K) download
From 3c53b86530a8c44d43579364b510114ee35cb200 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Mon, 27 Apr 2020 15:36:39 +0530
Subject: [PATCH v16 12/12] Add streaming option in pg_dump
---
src/bin/pg_dump/pg_dump.c | 9 +++++++--
src/bin/pg_dump/pg_dump.h | 1 +
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 5db4f57..11db7b7 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4210,6 +4210,7 @@ getSubscriptions(Archive *fout)
int i_oid;
int i_subname;
int i_rolname;
+ int i_substream;
int i_subconninfo;
int i_subslotname;
int i_subsynccommit;
@@ -4244,8 +4245,8 @@ getSubscriptions(Archive *fout)
appendPQExpBuffer(query,
"SELECT s.tableoid, s.oid, s.subname,"
"(%s s.subowner) AS rolname, "
- " s.subconninfo, s.subslotname, s.subsynccommit, "
- " s.subpublications "
+ " s.substream, s.subconninfo, s.subslotname, "
+ " s.subsynccommit, s.subpublications "
"FROM pg_subscription s "
"WHERE s.subdbid = (SELECT oid FROM pg_database"
" WHERE datname = current_database())",
@@ -4258,6 +4259,7 @@ getSubscriptions(Archive *fout)
i_oid = PQfnumber(res, "oid");
i_subname = PQfnumber(res, "subname");
i_rolname = PQfnumber(res, "rolname");
+ i_substream = PQfnumber(res, "substream");
i_subconninfo = PQfnumber(res, "subconninfo");
i_subslotname = PQfnumber(res, "subslotname");
i_subsynccommit = PQfnumber(res, "subsynccommit");
@@ -4274,6 +4276,7 @@ getSubscriptions(Archive *fout)
AssignDumpId(&subinfo[i].dobj);
subinfo[i].dobj.name = pg_strdup(PQgetvalue(res, i, i_subname));
subinfo[i].rolname = pg_strdup(PQgetvalue(res, i, i_rolname));
+ subinfo[i].substream = pg_strdup(PQgetvalue(res, i, i_substream));
subinfo[i].subconninfo = pg_strdup(PQgetvalue(res, i, i_subconninfo));
if (PQgetisnull(res, i, i_subslotname))
subinfo[i].subslotname = NULL;
@@ -4351,6 +4354,8 @@ dumpSubscription(Archive *fout, SubscriptionInfo *subinfo)
else
appendPQExpBufferStr(query, "NONE");
+ if (strcmp(subinfo->substream, "f") != 0)
+ appendPQExpBuffer(query, ", streaming = on");
if (strcmp(subinfo->subsynccommit, "off") != 0)
appendPQExpBuffer(query, ", synchronous_commit = %s", fmtId(subinfo->subsynccommit));
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 61c909e..5c5b072 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -623,6 +623,7 @@ typedef struct _SubscriptionInfo
{
DumpableObject dobj;
char *rolname;
+ char *substream;
char *subconninfo;
char *subslotname;
char *subsynccommit;
--
1.8.3.1
v16-0011-Provide-new-api-to-get-the-streaming-changes.patch (application/octet-stream, 4.7K) download
From af543b5a247e785cb8f4439fc89f979c2b5ec7b2 Mon Sep 17 00:00:00 2001
From: Dilip Kumar <dilipkumar@localhost.localdomain>
Date: Wed, 22 Apr 2020 16:33:07 +0530
Subject: [PATCH v16 11/12] Provide new api to get the streaming changes
---
src/backend/catalog/system_views.sql | 8 ++++++++
src/backend/replication/logical/logicalfuncs.c | 23 ++++++++++++++++++-----
src/include/catalog/pg_proc.dat | 9 +++++++++
3 files changed, 35 insertions(+), 5 deletions(-)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 65d650d..d9ab14b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1242,6 +1242,14 @@ LANGUAGE INTERNAL
VOLATILE ROWS 1000 COST 1000
AS 'pg_logical_slot_get_changes';
+CREATE OR REPLACE FUNCTION pg_logical_slot_get_streaming_changes(
+ IN slot_name name, IN upto_lsn pg_lsn, IN upto_nchanges int, VARIADIC options text[] DEFAULT '{}',
+ OUT lsn pg_lsn, OUT xid xid, OUT data text)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+VOLATILE ROWS 1000 COST 1000
+AS 'pg_logical_slot_get_streaming_changes';
+
CREATE OR REPLACE FUNCTION pg_logical_slot_peek_changes(
IN slot_name name, IN upto_lsn pg_lsn, IN upto_nchanges int, VARIADIC options text[] DEFAULT '{}',
OUT lsn pg_lsn, OUT xid xid, OUT data text)
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index f5384f1..7561141 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -108,7 +108,8 @@ check_permissions(void)
* Helper function for the various SQL callable logical decoding functions.
*/
static Datum
-pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool binary)
+pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm,
+ bool binary, bool streaming)
{
Name name;
XLogRecPtr upto_lsn;
@@ -237,6 +238,9 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
LogicalOutputPrepareWrite,
LogicalOutputWrite, NULL);
+ /* If called has not asked for streaming changes then disable it. */
+ ctx->streaming &= streaming;
+
MemoryContextSwitchTo(oldcontext);
/*
@@ -347,7 +351,16 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
Datum
pg_logical_slot_get_changes(PG_FUNCTION_ARGS)
{
- return pg_logical_slot_get_changes_guts(fcinfo, true, false);
+ return pg_logical_slot_get_changes_guts(fcinfo, true, false, false);
+}
+
+/*
+ * SQL function to get the streaming changes as text, consuming the data.
+ */
+Datum
+pg_logical_slot_get_streaming_changes(PG_FUNCTION_ARGS)
+{
+ return pg_logical_slot_get_changes_guts(fcinfo, true, false, true);
}
/*
@@ -356,7 +369,7 @@ pg_logical_slot_get_changes(PG_FUNCTION_ARGS)
Datum
pg_logical_slot_peek_changes(PG_FUNCTION_ARGS)
{
- return pg_logical_slot_get_changes_guts(fcinfo, false, false);
+ return pg_logical_slot_get_changes_guts(fcinfo, false, false, false);
}
/*
@@ -365,7 +378,7 @@ pg_logical_slot_peek_changes(PG_FUNCTION_ARGS)
Datum
pg_logical_slot_get_binary_changes(PG_FUNCTION_ARGS)
{
- return pg_logical_slot_get_changes_guts(fcinfo, true, true);
+ return pg_logical_slot_get_changes_guts(fcinfo, true, true, false);
}
/*
@@ -374,7 +387,7 @@ pg_logical_slot_get_binary_changes(PG_FUNCTION_ARGS)
Datum
pg_logical_slot_peek_binary_changes(PG_FUNCTION_ARGS)
{
- return pg_logical_slot_get_changes_guts(fcinfo, false, true);
+ return pg_logical_slot_get_changes_guts(fcinfo, false, true, false);
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9fb1ffe..3dfc5c1 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -10117,6 +10117,15 @@
proargmodes => '{i,i,i,v,o,o,o}',
proargnames => '{slot_name,upto_lsn,upto_nchanges,options,lsn,xid,data}',
prosrc => 'pg_logical_slot_get_binary_changes' },
+{ oid => '6150', descr => 'get streaming changes from replication slot',
+ proname => 'pg_logical_slot_get_streaming_changes', procost => '1000',
+ prorows => '1000', provariadic => 'text', proisstrict => 'f',
+ proretset => 't', provolatile => 'v', proparallel => 'u',
+ prorettype => 'record', proargtypes => 'name pg_lsn int4 _text',
+ proallargtypes => '{name,pg_lsn,int4,_text,pg_lsn,xid,text}',
+ proargmodes => '{i,i,i,v,o,o,o}',
+ proargnames => '{slot_name,upto_lsn,upto_nchanges,options,lsn,xid,data}',
+ prosrc => 'pg_logical_slot_get_streaming_changes' },
{ oid => '3784', descr => 'peek at changes from replication slot',
proname => 'pg_logical_slot_peek_changes', procost => '1000',
prorows => '1000', provariadic => 'text', proisstrict => 'f',
--
1.8.3.1
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-27 10:43 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-27 11:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-04-27 10:43 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> I have also fixed a couple of bugs internally reported by my colleague
> Neha Sharma.
>
I think it would be good if you can briefly explain what were the bugs
and how you fixed those?
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-27 10:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-04-27 11:26 ` Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 0 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-04-27 11:26 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Apr 27, 2020 at 4:13 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > I have also fixed a couple of bugs internally reported by my colleague
> > Neha Sharma.
> >
>
> I think it would be good if you can briefly explain what were the bugs
> and how you fixed those?
Issue1: If the concurrent transaction was aborted then in CATCH block
we were not freeing the memory of the toast_has, and it was causing
the assert that after the stream is complete txn->size != 0.
Issue2: After streaming is complete we set the txn->final_lsn and we
remember that in the local variable, But mistakenly it was remembered
in local TRY block variable so if there is a concurrent abort in the
CATCH block the variable value is always a zero. So after streaming
the final_lsn were becoming 0 and that was asserting.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-28 09:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-28 10:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Amit Kapila @ 2020-04-28 09:41 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
[latest patches]
v16-0004-Gracefully-handle-concurrent-aborts-of-uncommitt
- Any actions leading to transaction ID assignment are prohibited.
That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the
<literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment
are prohibited. That, among others,
..
@@ -1383,6 +1392,14 @@ heap_fetch(Relation relation,
bool valid;
/*
+ * We don't expect direct calls to heap_fetch with valid
+ * CheckXidAlive for regular tables. Track that below.
+ */
+ if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
+ !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
+ elog(ERROR, "unexpected heap_fetch call during logical decoding");
+
I think comments and code don't match. In the comment, we are saying
that via output plugins access to user catalog tables or regular
system catalog tables won't be allowed via heap_* APIs but code
doesn't seem to reflect it. I feel only
TransactionIdIsValid(CheckXidAlive) is sufficient here. See, the
original discussion about this point [1] (Refer "I think it'd also be
good to add assertions to codepaths not going through systable_*
asserting that ...").
Isn't it better to block the scan to user catalog tables or regular
system catalog tables for tableam scan APIs rather than at the heap
level? There might be some APIs like heap_getnext where such a check
might still be required but I guess it is still better to block at
tableam level.
[1] - https://www.postgresql.org/message-id/20180726200241.aje4dv4jsv25v4k2%40alap3.anarazel.de
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-28 09:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-04-28 10:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 03:18 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-29 09:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
0 siblings, 2 replies; 536+ messages in thread
From: Dilip Kumar @ 2020-04-28 10:25 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Apr 28, 2020 at 3:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> [latest patches]
>
> v16-0004-Gracefully-handle-concurrent-aborts-of-uncommitt
> - Any actions leading to transaction ID assignment are prohibited.
> That, among others,
> + Note that access to user catalog tables or regular system catalog tables
> + in the output plugins has to be done via the
> <literal>systable_*</literal> scan APIs only.
> + Access via the <literal>heap_*</literal> scan APIs will error out.
> + Additionally, any actions leading to transaction ID assignment
> are prohibited. That, among others,
> ..
> @@ -1383,6 +1392,14 @@ heap_fetch(Relation relation,
> bool valid;
>
> /*
> + * We don't expect direct calls to heap_fetch with valid
> + * CheckXidAlive for regular tables. Track that below.
> + */
> + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> + !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
> + elog(ERROR, "unexpected heap_fetch call during logical decoding");
> +
>
> I think comments and code don't match. In the comment, we are saying
> that via output plugins access to user catalog tables or regular
> system catalog tables won't be allowed via heap_* APIs but code
> doesn't seem to reflect it. I feel only
> TransactionIdIsValid(CheckXidAlive) is sufficient here. See, the
> original discussion about this point [1] (Refer "I think it'd also be
> good to add assertions to codepaths not going through systable_*
> asserting that ...").
Right, So I think we can just add an assert in these function that
Assert(!TransactionIdIsValid(CheckXidAlive)) ?
>
> Isn't it better to block the scan to user catalog tables or regular
> system catalog tables for tableam scan APIs rather than at the heap
> level? There might be some APIs like heap_getnext where such a check
> might still be required but I guess it is still better to block at
> tableam level.
>
> [1] - https://www.postgresql.org/message-id/20180726200241.aje4dv4jsv25v4k2%40alap3.anarazel.de
Okay, let me analyze this part. Because someplace we have to keep at
heap level like heap_getnext and other places at tableam level so it
seems a bit inconsistent. Also, I think the number of checks might
going to increase because some of the heap functions like
heap_hot_search_buffer are being called from multiple tableam calls,
so we need to put check at every place.
Another point is that I feel some of the checks what we have today
might not be required like heap_finish_speculative, is not fetching
any tuple for us so why do we need to care about this function?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-28 09:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-28 10:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-29 03:18 ` Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 0 replies; 536+ messages in thread
From: Amit Kapila @ 2020-04-29 03:18 UTC (permalink / raw)
To: Dilip Kumar <dilipbalaut@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Apr 28, 2020 at 3:55 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Apr 28, 2020 at 3:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > [latest patches]
> >
> > v16-0004-Gracefully-handle-concurrent-aborts-of-uncommitt
> > - Any actions leading to transaction ID assignment are prohibited.
> > That, among others,
> > + Note that access to user catalog tables or regular system catalog tables
> > + in the output plugins has to be done via the
> > <literal>systable_*</literal> scan APIs only.
> > + Access via the <literal>heap_*</literal> scan APIs will error out.
> > + Additionally, any actions leading to transaction ID assignment
> > are prohibited. That, among others,
> > ..
> > @@ -1383,6 +1392,14 @@ heap_fetch(Relation relation,
> > bool valid;
> >
> > /*
> > + * We don't expect direct calls to heap_fetch with valid
> > + * CheckXidAlive for regular tables. Track that below.
> > + */
> > + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> > + !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
> > + elog(ERROR, "unexpected heap_fetch call during logical decoding");
> > +
> >
> > I think comments and code don't match. In the comment, we are saying
> > that via output plugins access to user catalog tables or regular
> > system catalog tables won't be allowed via heap_* APIs but code
> > doesn't seem to reflect it. I feel only
> > TransactionIdIsValid(CheckXidAlive) is sufficient here. See, the
> > original discussion about this point [1] (Refer "I think it'd also be
> > good to add assertions to codepaths not going through systable_*
> > asserting that ...").
>
> Right, So I think we can just add an assert in these function that
> Assert(!TransactionIdIsValid(CheckXidAlive)) ?
>
I am fine with Assertion but update the documentation accordingly.
However, I think you can once cross-verify if there are any output
plugins that are already using such APIs. There is a list of "Logical
Decoding Plugins" on the wiki [1], just look into those once.
> >
> > Isn't it better to block the scan to user catalog tables or regular
> > system catalog tables for tableam scan APIs rather than at the heap
> > level? There might be some APIs like heap_getnext where such a check
> > might still be required but I guess it is still better to block at
> > tableam level.
> >
> > [1] - https://www.postgresql.org/message-id/20180726200241.aje4dv4jsv25v4k2%40alap3.anarazel.de
>
> Okay, let me analyze this part. Because someplace we have to keep at
> heap level like heap_getnext and other places at tableam level so it
> seems a bit inconsistent. Also, I think the number of checks might
> going to increase because some of the heap functions like
> heap_hot_search_buffer are being called from multiple tableam calls,
> so we need to put check at every place.
>
> Another point is that I feel some of the checks what we have today
> might not be required like heap_finish_speculative, is not fetching
> any tuple for us so why do we need to care about this function?
>
Yeah, I don't see the need for such a check (or Assertion) in
heap_finish_speculative.
One additional comment:
---------------------------------------
- Any actions leading to transaction ID assignment are prohibited.
That, among others,
+ Note that access to user catalog tables or regular system catalog tables
+ in the output plugins has to be done via the
<literal>systable_*</literal> scan APIs only.
+ Access via the <literal>heap_*</literal> scan APIs will error out.
+ Additionally, any actions leading to transaction ID assignment
are prohibited. That, among others,
The above text doesn't seem to be aligned properly and you need to
update it if we want to change the error to Assertion for heap APIs
[1] - https://wiki.postgresql.org/wiki/Logical_Decoding_Plugins
--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-28 09:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-28 10:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-29 09:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 09:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-04-29 09:26 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Tue, Apr 28, 2020 at 3:55 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Apr 28, 2020 at 3:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > [latest patches]
> >
> > v16-0004-Gracefully-handle-concurrent-aborts-of-uncommitt
> > - Any actions leading to transaction ID assignment are prohibited.
> > That, among others,
> > + Note that access to user catalog tables or regular system catalog tables
> > + in the output plugins has to be done via the
> > <literal>systable_*</literal> scan APIs only.
> > + Access via the <literal>heap_*</literal> scan APIs will error out.
> > + Additionally, any actions leading to transaction ID assignment
> > are prohibited. That, among others,
> > ..
> > @@ -1383,6 +1392,14 @@ heap_fetch(Relation relation,
> > bool valid;
> >
> > /*
> > + * We don't expect direct calls to heap_fetch with valid
> > + * CheckXidAlive for regular tables. Track that below.
> > + */
> > + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> > + !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
> > + elog(ERROR, "unexpected heap_fetch call during logical decoding");
> > +
> >
> > I think comments and code don't match. In the comment, we are saying
> > that via output plugins access to user catalog tables or regular
> > system catalog tables won't be allowed via heap_* APIs but code
> > doesn't seem to reflect it. I feel only
> > TransactionIdIsValid(CheckXidAlive) is sufficient here. See, the
> > original discussion about this point [1] (Refer "I think it'd also be
> > good to add assertions to codepaths not going through systable_*
> > asserting that ...").
>
> Right, So I think we can just add an assert in these function that
> Assert(!TransactionIdIsValid(CheckXidAlive)) ?
>
> >
> > Isn't it better to block the scan to user catalog tables or regular
> > system catalog tables for tableam scan APIs rather than at the heap
> > level? There might be some APIs like heap_getnext where such a check
> > might still be required but I guess it is still better to block at
> > tableam level.
> >
> > [1] - https://www.postgresql.org/message-id/20180726200241.aje4dv4jsv25v4k2%40alap3.anarazel.de
>
> Okay, let me analyze this part. Because someplace we have to keep at
> heap level like heap_getnext and other places at tableam level so it
> seems a bit inconsistent. Also, I think the number of checks might
> going to increase because some of the heap functions like
> heap_hot_search_buffer are being called from multiple tableam calls,
> so we need to put check at every place.
>
> Another point is that I feel some of the checks what we have today
> might not be required like heap_finish_speculative, is not fetching
> any tuple for us so why do we need to care about this function?
While testing these changes, I have noticed that the systable_* APIs
internally, calls tableam apis and so if we just put assert
Assert(!TransactionIdIsValid(CheckXidAlive)) then it will always hit
that assert. Whether we put these assert in heap APIs or the tableam
APIs because systable_ always access heap through tableam APIs.
Refer below callstack
#0 table_index_fetch_tuple (scan=0x2392558, tid=0x2392270,
snapshot=0x2392178, slot=0x2391f60, call_again=0x2392276,
all_dead=0x7fff4b6cc89e)
at ../../../../src/include/access/tableam.h:1035
#1 0x00000000005100b6 in index_fetch_heap (scan=0x2392210,
slot=0x2391f60) at indexam.c:577
#2 0x00000000005101ea in index_getnext_slot (scan=0x2392210,
direction=ForwardScanDirection, slot=0x2391f60) at indexam.c:637
#3 0x000000000050e8f9 in systable_getnext (sysscan=0x2391f08) at genam.c:474
#4 0x0000000000aa44a2 in RelidByRelfilenode (reltablespace=0,
relfilenode=16593) at relfilenodemap.c:213
#5 0x00000000008a64da in ReorderBufferProcessTXN (rb=0x23734b0,
txn=0x2398e28, commit_lsn=23953600, snapshot_now=0x237b168,
command_id=0, streaming=false)
at reorderbuffer.c:1823
#6 0x00000000008a7201 in ReorderBufferCommit (rb=0x23734b0, xid=518,
commit_lsn=23953600, end_lsn=23953648, commit_time=641466886013448,
origin_id=0, origin_lsn=0)
at reorderbuffer.c:2315
#7 0x00000000008985b1 in DecodeCommit (ctx=0x22e16a0,
buf=0x7fff4b6cce30, parsed=0x7fff4b6ccca0, xid=518) at decode.c:654
#8 0x0000000000897a76 in DecodeXactOp (ctx=0x22e16a0,
buf=0x7fff4b6cce30) at decode.c:261
#9 0x0000000000897739 in LogicalDecodingProcessRecord (ctx=0x22e16a0,
record=0x22e19a0) at decode.c:130
So basically, the problem is that we can not distinguish whether the
tableam/heap routine is called directly or via systable_*.
Now I understand the current code was actually giving error for the
user table not the system table with the assumption that the system
table will come to this function only via systable_*. Only user table
can come directly. So if this is not a system table i.e. we reach
here directly so error out. Now, I am not sure if it is not for the
system table then what is the purpose of throwing that error?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-28 09:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-28 10:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 09:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
@ 2020-04-29 09:49 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-30 07:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-04-29 09:49 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Wed, Apr 29, 2020 at 2:56 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Apr 28, 2020 at 3:55 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> > On Tue, Apr 28, 2020 at 3:11 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Mon, Apr 27, 2020 at 4:05 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > >
> > > [latest patches]
> > >
> > > v16-0004-Gracefully-handle-concurrent-aborts-of-uncommitt
> > > - Any actions leading to transaction ID assignment are prohibited.
> > > That, among others,
> > > + Note that access to user catalog tables or regular system catalog tables
> > > + in the output plugins has to be done via the
> > > <literal>systable_*</literal> scan APIs only.
> > > + Access via the <literal>heap_*</literal> scan APIs will error out.
> > > + Additionally, any actions leading to transaction ID assignment
> > > are prohibited. That, among others,
> > > ..
> > > @@ -1383,6 +1392,14 @@ heap_fetch(Relation relation,
> > > bool valid;
> > >
> > > /*
> > > + * We don't expect direct calls to heap_fetch with valid
> > > + * CheckXidAlive for regular tables. Track that below.
> > > + */
> > > + if (unlikely(TransactionIdIsValid(CheckXidAlive) &&
> > > + !(IsCatalogRelation(relation) || RelationIsUsedAsCatalogTable(relation))))
> > > + elog(ERROR, "unexpected heap_fetch call during logical decoding");
> > > +
> > >
> > > I think comments and code don't match. In the comment, we are saying
> > > that via output plugins access to user catalog tables or regular
> > > system catalog tables won't be allowed via heap_* APIs but code
> > > doesn't seem to reflect it. I feel only
> > > TransactionIdIsValid(CheckXidAlive) is sufficient here. See, the
> > > original discussion about this point [1] (Refer "I think it'd also be
> > > good to add assertions to codepaths not going through systable_*
> > > asserting that ...").
> >
> > Right, So I think we can just add an assert in these function that
> > Assert(!TransactionIdIsValid(CheckXidAlive)) ?
> >
> > >
> > > Isn't it better to block the scan to user catalog tables or regular
> > > system catalog tables for tableam scan APIs rather than at the heap
> > > level? There might be some APIs like heap_getnext where such a check
> > > might still be required but I guess it is still better to block at
> > > tableam level.
> > >
> > > [1] - https://www.postgresql.org/message-id/20180726200241.aje4dv4jsv25v4k2%40alap3.anarazel.de
> >
> > Okay, let me analyze this part. Because someplace we have to keep at
> > heap level like heap_getnext and other places at tableam level so it
> > seems a bit inconsistent. Also, I think the number of checks might
> > going to increase because some of the heap functions like
> > heap_hot_search_buffer are being called from multiple tableam calls,
> > so we need to put check at every place.
> >
> > Another point is that I feel some of the checks what we have today
> > might not be required like heap_finish_speculative, is not fetching
> > any tuple for us so why do we need to care about this function?
>
> While testing these changes, I have noticed that the systable_* APIs
> internally, calls tableam apis and so if we just put assert
> Assert(!TransactionIdIsValid(CheckXidAlive)) then it will always hit
> that assert. Whether we put these assert in heap APIs or the tableam
> APIs because systable_ always access heap through tableam APIs.
>
> Refer below callstack
> #0 table_index_fetch_tuple (scan=0x2392558, tid=0x2392270,
> snapshot=0x2392178, slot=0x2391f60, call_again=0x2392276,
> all_dead=0x7fff4b6cc89e)
> at ../../../../src/include/access/tableam.h:1035
> #1 0x00000000005100b6 in index_fetch_heap (scan=0x2392210,
> slot=0x2391f60) at indexam.c:577
> #2 0x00000000005101ea in index_getnext_slot (scan=0x2392210,
> direction=ForwardScanDirection, slot=0x2391f60) at indexam.c:637
> #3 0x000000000050e8f9 in systable_getnext (sysscan=0x2391f08) at genam.c:474
> #4 0x0000000000aa44a2 in RelidByRelfilenode (reltablespace=0,
> relfilenode=16593) at relfilenodemap.c:213
> #5 0x00000000008a64da in ReorderBufferProcessTXN (rb=0x23734b0,
> txn=0x2398e28, commit_lsn=23953600, snapshot_now=0x237b168,
> command_id=0, streaming=false)
> at reorderbuffer.c:1823
> #6 0x00000000008a7201 in ReorderBufferCommit (rb=0x23734b0, xid=518,
> commit_lsn=23953600, end_lsn=23953648, commit_time=641466886013448,
> origin_id=0, origin_lsn=0)
> at reorderbuffer.c:2315
> #7 0x00000000008985b1 in DecodeCommit (ctx=0x22e16a0,
> buf=0x7fff4b6cce30, parsed=0x7fff4b6ccca0, xid=518) at decode.c:654
> #8 0x0000000000897a76 in DecodeXactOp (ctx=0x22e16a0,
> buf=0x7fff4b6cce30) at decode.c:261
> #9 0x0000000000897739 in LogicalDecodingProcessRecord (ctx=0x22e16a0,
> record=0x22e19a0) at decode.c:130
>
> So basically, the problem is that we can not distinguish whether the
> tableam/heap routine is called directly or via systable_*.
>
> Now I understand the current code was actually giving error for the
> user table not the system table with the assumption that the system
> table will come to this function only via systable_*. Only user table
> can come directly. So if this is not a system table i.e. we reach
> here directly so error out. Now, I am not sure if it is not for the
> system table then what is the purpose of throwing that error?
Putting some more thought upon this, I am just wondering what do we
really want any such check because, we are always getting relation
description from the reorder buffer code, not from the pgoutput
plugin. And, our main issue with the concurrent abort is that we
shall not get the wrong catalog entry for decoding our tuple. So if
we are always getting our relation entry using RelationIdGetRelation
then why should we bother about how output plugin is accessing
system/user relations?
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
* Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-13 12:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 09:40 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-30 10:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-02-11 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 06:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-28 09:41 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-04-28 10:25 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 09:26 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 09:49 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-04-30 07:00 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
2020-05-01 15:10 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Dilip Kumar <dilipbalaut@gmail.com>
2020-05-04 11:46 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
@ 2020-05-05 03:57 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-05 04:54 ` Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 536+ messages in thread
From: Dilip Kumar @ 2020-05-05 03:57 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Kuntal Ghosh <kuntalghosh.2007@gmail.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; pgsql-hackers
On Mon, May 4, 2020 at 5:16 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, May 1, 2020 at 8:41 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >
> 5. Shouldn't we add a check in table_scan_sample_next_block and
> table_scan_sample_next_tuple APIs as well?
I am not sure that we need to do that, Because generally, we want to
avoid getting any wrong system table tuple which we can use for taking
some decision or decode tuple. But, I don't think that
table_scan_sample falls under that category.
> > Apart from this, I have also fixed one defect raised by my colleague
> > Neha Sharma. That issue is the incomplete toast tuple flag was not
> > reset when the main table tuple was inserted through speculative
> > insert and due to that data was not streamed even if later we were
> > getting speculative confirm because incomplete toast flag was never
> > reset. This patch also includes the fix for the issue raised by Erik.
> >
>
> It would be better if you can mention which all patches contain the
> changes as it will be easier to review the fix.
Fix1: v17-0010-Bugfix-handling-of-incomplete-toast-tuple.patch
Fix2: patch: v17-0002-Issue-individual-invalidations-with-wal_level-lo.patch
I will work on other comments and send the updated patch.
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 536+ messages in thread
349 further messages in this thread omitted from this page (render size limit).
Use the per-message pages, or the t.mbox.gz byte stream, for the complete thread.
end of thread, other threads:[~2020-08-28 10:16 UTC | newest]
Thread overview: 536+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-12-23 04:57 PATCH: logical_work_mem and logical streaming of large in-progress transactions Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 14:03 ` Erikjan Rijkers <er@xs4all.nl>
2017-12-23 20:06 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-23 22:23 ` Erik Rijkers <er@xs4all.nl>
2017-12-24 00:42 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-24 09:00 ` Erik Rijkers <er@xs4all.nl>
2017-12-25 17:40 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2017-12-25 22:08 ` Erik Rijkers <er@xs4all.nl>
2017-12-26 14:50 ` Dmitry Dolgov <9erthalion6@gmail.com>
2017-12-24 04:51 ` Craig Ringer <craig@2ndquadrant.com>
2017-12-24 13:43 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-02 15:07 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-03 19:53 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:24 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-27 09:03 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-27 11:25 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-28 08:06 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 19:09 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-29 05:54 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-29 17:30 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-29 17:56 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 13:25 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-01 13:51 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-01 22:57 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-02 22:32 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-03 07:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-10 10:27 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:16 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-22 05:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:22 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-22 19:02 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-10-23 10:50 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-30 04:08 ` vignesh C <vignesh21@gmail.com>
2019-10-30 07:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 09:13 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-04 10:02 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-04 10:05 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-12 10:42 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-11-12 11:34 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2019-11-04 11:51 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-05 05:38 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-06 11:03 ` vignesh C <vignesh21@gmail.com>
2019-11-13 12:25 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 04:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-14 06:40 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-14 10:09 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:20 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-15 10:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-15 10:49 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-18 11:32 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-19 11:53 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-20 05:45 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-20 14:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-21 03:32 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-22 07:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-01 02:28 ` Michael Paquier <michael@paquier.xyz>
2019-12-02 08:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-09 07:56 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-11 11:52 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-12 04:14 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-13 10:05 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-30 09:40 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 10:37 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 03:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 08:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 10:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-06 11:06 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-06 11:14 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-08 07:42 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 04:05 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 05:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-09 06:39 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-09 07:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-13 09:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-14 05:13 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-22 05:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 05:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 06:12 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 06:28 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 08:00 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-28 08:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-28 09:06 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-04 05:29 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 03:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:18 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 04:44 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 21:35 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-10 21:37 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-11 02:48 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-14 05:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-21 16:36 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-01-30 10:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-30 12:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-31 02:38 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:15 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-05 10:35 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-07 10:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-10 08:22 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-10 09:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-11 03:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-13 03:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-29 05:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-03 21:46 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-03 21:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-04 03:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-05 17:55 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-28 06:25 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-28 08:49 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-28 09:59 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 00:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-29 05:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-29 15:31 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-30 06:17 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-30 15:27 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-07 06:47 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-08 00:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-09 09:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 10:44 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 11:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 12:41 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-13 13:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-13 18:13 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-14 10:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 15:44 ` Erik Rijkers <er@xs4all.nl>
2020-04-15 04:44 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:33 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 09:46 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 09:07 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 09:10 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 12:42 ` Erik Rijkers <er@xs4all.nl>
2020-04-18 14:01 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-21 12:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 14:49 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-22 16:01 ` Erik Rijkers <er@xs4all.nl>
2020-04-23 03:24 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-23 08:58 ` Erik Rijkers <er@xs4all.nl>
2020-04-24 06:24 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 05:45 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-04-29 07:07 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-04-29 07:20 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 20:10 ` Kuntal Ghosh <kuntalghosh.2007@gmail.com>
2020-04-27 10:34 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-27 10:43 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-27 11:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-28 09:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-28 10:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 03:18 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-29 09:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-29 09:49 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-30 07:00 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-01 15:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-04 11:46 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-05 03:57 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-05 04:54 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-05 05:01 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-05 05:31 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-05 10:36 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-05 13:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-07 12:46 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-12 11:09 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-13 06:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-13 11:20 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-13 15:46 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-14 05:41 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-15 09:17 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-19 12:31 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-22 06:24 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-22 11:16 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-25 14:37 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-25 14:37 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-25 15:18 ` Erik Rijkers <er@xs4all.nl>
2020-05-26 02:15 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-27 14:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-28 11:52 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-02 10:28 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-02 11:25 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-02 14:22 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-03 09:13 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-04 08:35 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-07 11:36 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-10 09:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-10 10:30 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-10 11:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-10 11:40 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-12 06:08 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-12 11:04 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-15 03:41 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-15 12:59 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-16 09:06 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-17 10:26 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-22 06:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-24 10:40 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-22 06:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-29 09:24 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-29 15:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-03 10:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-05 06:07 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-07 11:37 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-08 06:23 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-09 09:34 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-16 14:19 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-17 04:02 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-17 04:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-18 15:32 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-18 15:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-22 10:56 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-22 11:00 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-29 10:54 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-30 03:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-30 04:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-30 11:50 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-04 06:04 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-05 11:17 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-06 06:01 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-06 06:13 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-06 09:39 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-07 15:29 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-12 16:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 05:17 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 06:00 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-15 04:05 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-15 11:21 ` Ajin Cherian <itsajin@gmail.com>
2020-07-15 11:23 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 10:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 04:43 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-13 05:09 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 05:39 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-13 09:02 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 09:26 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-13 09:34 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-13 10:30 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-13 10:38 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-14 12:09 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-15 03:59 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-15 13:29 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-16 06:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-16 10:55 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-20 06:31 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-20 08:45 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-20 11:11 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-20 13:15 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-21 03:37 ` Ajin Cherian <itsajin@gmail.com>
2020-07-22 03:48 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-22 04:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-22 10:50 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-22 11:24 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-23 06:01 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-24 11:35 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-24 13:47 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-25 11:38 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-26 05:34 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-26 13:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-28 04:21 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-29 05:16 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-30 06:57 ` Ajin Cherian <itsajin@gmail.com>
2020-07-30 09:53 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-31 09:52 ` Ajin Cherian <itsajin@gmail.com>
2020-08-04 04:42 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-04 07:11 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-05 12:55 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-05 14:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-06 09:13 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-07 08:34 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-13 06:38 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-13 13:17 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-15 10:02 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-17 12:58 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-19 04:40 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-19 06:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-19 08:05 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-20 08:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-20 08:59 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-20 12:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-21 03:44 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-21 03:58 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-21 04:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-21 05:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-21 05:03 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-21 09:43 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-21 11:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-22 03:08 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-24 16:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-25 04:01 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-25 05:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-25 12:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-25 14:05 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-26 17:52 ` Jeff Janes <jeff.janes@gmail.com>
2020-08-27 05:45 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-28 08:48 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-28 10:16 ` Neha Sharma <neha.sharma@enterprisedb.com>
2020-08-19 08:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-08-14 04:40 ` Thomas Munro <thomas.munro@gmail.com>
2020-08-14 06:14 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-14 22:43 ` Thomas Munro <thomas.munro@gmail.com>
2020-08-15 04:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-08-15 10:02 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-15 04:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-05 11:05 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-05 15:07 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-08 04:06 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-08 10:02 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-08 14:01 ` Ajin Cherian <itsajin@gmail.com>
2020-07-09 02:27 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-09 02:48 ` Ajin Cherian <itsajin@gmail.com>
2020-07-09 03:17 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-09 03:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-09 03:30 ` Amit Kapila <amit.kapila16@gmail.com>
2020-07-10 03:51 ` Ajin Cherian <itsajin@gmail.com>
2020-07-10 05:11 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-10 05:31 ` Ajin Cherian <itsajin@gmail.com>
2020-07-10 06:18 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-09 09:11 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-07-09 11:15 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-22 11:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-22 11:55 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-22 13:08 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-23 02:48 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-23 04:43 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-23 13:30 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-24 10:34 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-24 10:57 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-24 11:08 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-25 13:40 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-26 05:09 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-26 05:51 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-26 06:17 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-28 15:26 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-06-28 15:25 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-26 09:34 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-28 07:27 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-28 09:45 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-28 10:09 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-29 15:01 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-22 12:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-15 09:17 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-15 10:34 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-15 10:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-15 11:05 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-15 11:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-17 07:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-18 10:40 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-18 12:27 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-19 09:04 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-19 10:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-19 11:02 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-22 12:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-22 12:51 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-26 04:57 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-26 09:13 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-26 11:16 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-27 11:49 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-05-29 10:22 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-04 11:35 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-06-09 10:09 ` Amit Kapila <amit.kapila16@gmail.com>
2020-06-09 10:22 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-29 15:01 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-22 12:51 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-26 06:30 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-28 07:16 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-28 09:11 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-28 10:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-29 15:01 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-05-19 12:04 ` Amit Kapila <amit.kapila16@gmail.com>
2020-05-22 12:52 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 09:27 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-14 10:11 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-14 10:27 ` Amit Kapila <amit.kapila16@gmail.com>
2020-04-14 10:36 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-04-16 21:25 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-04-14 06:34 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-04 04:58 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-04 09:03 ` Amit Kapila <amit.kapila16@gmail.com>
2020-03-04 09:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-03-05 17:50 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2020-03-06 04:49 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-01 05:35 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-05 04:12 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-06 09:21 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-10 04:51 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-22 05:10 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-22 16:37 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2020-01-23 03:37 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-10 04:22 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-10 04:53 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-28 16:03 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-12-29 08:04 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-12-30 10:13 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-02 05:50 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 05:22 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-02-03 04:21 ` Amit Kapila <amit.kapila16@gmail.com>
2020-02-04 05:20 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-04 04:30 ` Amit Kapila <amit.kapila16@gmail.com>
2020-01-04 06:24 ` Dilip Kumar <dilipbalaut@gmail.com>
2020-01-10 20:56 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-12-11 18:16 ` Robert Haas <robertmhaas@gmail.com>
2019-12-12 11:41 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-24 05:28 ` Robert Haas <robertmhaas@gmail.com>
2020-01-03 09:24 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-13 09:16 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-20 08:29 ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2019-12-20 13:42 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-20 06:17 ` Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
2019-12-20 13:30 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-24 05:46 ` Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
2019-12-24 08:21 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-26 07:05 ` Masahiko Sawada <masahiko.sawada@2ndquadrant.com>
2019-12-30 10:19 ` Amit Kapila <amit.kapila16@gmail.com>
2019-12-22 11:34 ` vignesh C <vignesh21@gmail.com>
2019-12-24 04:24 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-03 09:13 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-13 06:54 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-13 11:46 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-14 01:21 ` Craig Ringer <craig@2ndquadrant.com>
2019-10-14 02:11 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-14 09:39 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-18 12:01 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-21 05:18 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-21 09:20 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-21 09:41 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 05:00 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-10-22 17:12 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-10-23 09:13 ` Amit Kapila <amit.kapila16@gmail.com>
2019-10-24 13:37 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-04 10:16 ` vignesh C <vignesh21@gmail.com>
2019-11-06 06:03 ` vignesh C <vignesh21@gmail.com>
2019-11-07 09:49 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-07 10:19 ` Dilip Kumar <dilipbalaut@gmail.com>
2019-11-07 11:43 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-16 13:14 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-19 11:55 ` Amit Kapila <amit.kapila16@gmail.com>
2019-11-21 10:44 ` Amit Kapila <amit.kapila16@gmail.com>
2018-01-03 20:06 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-03 20:13 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-09 02:36 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-19 14:34 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-19 22:08 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-31 06:53 ` Masahiko Sawada <sawada.mshk@gmail.com>
2018-02-01 22:50 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-03 00:55 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-03 05:19 ` Erik Rijkers <er@xs4all.nl>
2018-03-03 14:52 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-09 16:07 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-07-01 11:43 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-12-16 14:31 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-12-16 17:54 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-12-17 16:23 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2018-12-17 22:28 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-12-18 14:07 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2018-12-18 23:56 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-12-19 09:58 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-01-14 18:23 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-02-04 05:51 ` Michael Paquier <michael@paquier.xyz>
2019-02-04 13:49 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-08-28 17:17 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-08-28 19:06 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-08-29 14:37 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-08-29 18:48 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-08-30 15:59 ` Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2019-09-16 16:54 ` Alexey Kondratov <a.kondratov@postgrespro.ru>
2019-09-16 19:29 ` Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2019-09-16 20:57 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-10-02 04:59 ` Michael Paquier <michael@paquier.xyz>
2018-01-11 19:41 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-11 23:23 ` Greg Stark <stark@mit.edu>
2018-01-12 16:35 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-01-13 04:19 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-18 13:24 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-02-01 14:51 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2018-02-01 22:51 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-02 01:12 ` Andres Freund <andres@anarazel.de>
2018-03-02 02:33 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-02 02:39 ` David Steele <david@pgmasters.net>
2018-03-02 20:05 ` Andres Freund <andres@anarazel.de>
2018-03-03 01:00 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-03 01:01 ` Andres Freund <andres@anarazel.de>
2018-03-03 01:34 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-03 01:36 ` Andres Freund <andres@anarazel.de>
2018-03-03 01:37 ` David Steele <david@pgmasters.net>
2018-03-03 01:54 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-03 02:08 ` David Steele <david@pgmasters.net>
2018-03-02 20:06 ` Robert Haas <robertmhaas@gmail.com>
2018-03-02 20:21 ` David Steele <david@pgmasters.net>
2018-03-02 21:06 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-01-13 04:01 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2018-03-29 16:34 ` Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2019-09-02 22:06 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-03 10:39 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-26 13:28 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-26 18:36 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-27 05:48 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-25 13:25 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-26 18:07 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-26 19:33 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 19:36 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-09-26 21:24 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-26 21:46 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-09-27 09:40 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-28 12:03 ` Amit Kapila <amit.kapila16@gmail.com>
2019-09-29 04:34 ` Dilip Kumar <dilipbalaut@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox